From Stateless to Context-Engineered

The Goldfish Problem

Imagine you're at a coffee shop talking to a brilliant consultant. You explain your entire business problem — the architecture, the constraints, the deadline. They give perfect advice. You come back the next morning, and they greet you like they've never seen you before. Every conversation starts from absolute zero.

That's how LLMs work. Not because they're poorly designed, but because of a fundamental architectural choice: every API call is completely isolated. When you send a message to GPT-4o, Claude, or any other LLM, the model processes your input, generates a response, and then — poof — forgets the entire interaction ever happened.

There's no internal notebook. No session state. No "hey, we talked about this yesterday." The model is stateless in the strictest sense: it has zero memory between calls.

This isn't a bug — it's a design choice. LLMs are token-prediction machines: they take a sequence of input tokens, process them through billions of parameters, and output the next most likely tokens. Once the response is generated, all that processing is discarded. The model doesn't "learn" from your conversation or store it anywhere.

If you've used ChatGPT or Claude and felt like they "remember" your conversation, that's because the application layer (the chat interface) is doing the remembering, not the model. Behind the scenes, the app stores your conversation history and sends the entire thing back to the model with every new message. The model isn't remembering — it's being reminded.

"Just Send Everything"

The simplest solution to statelessness? Keep a growing list of every message in the conversation, and send all of them to the model every single time.

Think of it like recording every meeting on your team. When you need context for a new discussion, you play back all previous recordings first. For the first few meetings, this works perfectly. By meeting 50, you're spending more time watching recordings than having actual discussions.

This approach — appending messages and sending everything — is how most chatbots start. You maintain an array of messages (system prompt + conversation turns), and each API call sends the entire array.

For short conversations — say 5 to 15 messages — this works fine. The model sees the full context and gives coherent, contextual responses. But there's a hard ceiling on how much you can send, and it's called the context window.

The Math That Matters: Context Windows

The context window is the total amount of information a model can process in a single call — both the input you send and the output it generates. It's measured in tokens, where one token is roughly three-quarters of a word (or about four characters). So 1,000 tokens is approximately 750 English words.

Think of the context window as a desk. Everything the model can "see" has to fit on this desk — your system prompt, the entire conversation history, any retrieved documents, and the response it's generating. If your papers overflow the desk, they fall off the edge and disappear.

You might look at Gemini 2.5's million-token window and think: "Problem solved. Just dump everything in." But three forces conspire against you:

1. Cost scales linearly with tokens. Sending 100K tokens every call is 25x more expensive than sending 4K. For high-volume applications, this adds up fast.
2. Latency increases with context size. The model has to process every token you send. More tokens means slower responses — sometimes painfully so.
3. Performance degrades with length. This is the counterintuitive one. Research consistently shows that models lose track of information in long contexts, especially information placed in the middle. We'll explore this deeply in Part 3.

So even with massive context windows, you need strategies to manage what goes in. The two most fundamental strategies are trimming and summarization.

Strategy 1: Trimming — First In, First Out

The simplest strategy: when the conversation gets too long, drop the oldest messages. Keep the system prompt (which defines the AI's behavior) and the most recent messages. Everything else falls off the edge.

Think of a conveyor belt at an airport baggage claim. New bags arrive from one end, and once the belt is full, the oldest bags drop off the other end. Your system prompt is the one bag that's bolted to the belt — it never moves. Everything else is first in, first out.

Trimming is fast, cheap, and easy to implement. You count tokens from the newest message backward until you hit the limit, and everything older gets cut. No extra API calls, no processing overhead.

But trimming has a critical weakness: it has no concept of importance. A user mentioning their name on message 1 is just as likely to be trimmed as a casual "sounds good" on message 3. A patient's drug allergy mentioned early in a conversation can be silently dropped when newer, less important messages push it out.

For some applications — casual chat, quick Q&A sessions, interactions under 20 messages — trimming works perfectly. For anything involving safety-critical information, long-running tasks, or users who expect continuity, you need something smarter.

def trim_messages(messages, max_tokens=4000, model="gpt-4o"):
    """Keep the system prompt + most recent messages that fit."""
    import tiktoken
    enc = tiktoken.encoding_for_model(model)

    # Always keep the system prompt
    system_msg = messages[0]
    system_tokens = len(enc.encode(system_msg["content"]))

    # Add messages from newest to oldest until we hit the limit
    trimmed = []
    token_count = system_tokens

    for msg in reversed(messages[1:]):
        msg_tokens = len(enc.encode(msg["content"]))
        if token_count + msg_tokens > max_tokens:
            break
        trimmed.insert(0, msg)
        token_count += msg_tokens

    return [system_msg] + trimmed

def smart_trim(messages, max_tokens=4000):
    """Trim with priority: pinned > recent > old."""
    system = messages[0]
    pinned = [m for m in messages[1:] if m.get("pinned")]
    regular = [m for m in messages[1:] if not m.get("pinned")]

    # Start with system + pinned (always kept)
    result = [system] + pinned
    token_count = count_tokens(result)

    # Add regular messages from newest to oldest
    for msg in reversed(regular):
        msg_tokens = count_tokens([msg])
        if token_count + msg_tokens > max_tokens:
            break
        result.insert(len([system] + pinned), msg)
        token_count += msg_tokens

    return result

Strategy 2: Summarization — Compress, Don't Delete

What if instead of throwing away old messages, you condensed them into a summary? Rather than losing that the user mentioned their name, their project requirements, and their deadline, you compress 20 messages into a single paragraph that captures the essential facts.

Think of it as the difference between keeping every receipt from a year of grocery shopping versus keeping a monthly budget spreadsheet. The receipts have all the detail, but the spreadsheet tells you everything you actually need to know.

The approach: when the conversation exceeds a threshold, take the oldest messages, send them to a (usually cheaper) model with a summarization prompt, and replace them with the resulting summary. The model now sees: system prompt → summary of old messages → last 6 recent messages.

Summarization preserves intent where trimming destroys it. The summary captures "the user is Bahgat, building a FastAPI service with JWT authentication, deadline is March" even though those details were scattered across messages 1 through 15. With trimming, all of that would be gone once message 20 arrives.

The tradeoff? Summarization costs extra. Each summarization step requires an additional API call (typically using a cheaper model like GPT-4o-mini). It adds 300–800ms of latency. And summarization is lossy — you inevitably lose some nuance and detail. A summary might capture "the user has a food allergy" but drop the specific allergen. For casual conversations that's fine. For medical applications, it's dangerous.

But there's not just one way to summarize. In practice, there are three distinct strategies, and the right choice depends on your use case:

For short conversations (under 20 turns), stacking summaries works fine — the stack stays small. For longer interactions, batch or rolling summaries are necessary. The single rolling summary is the most aggressive: it never grows, but it risks overwriting important early details if the conversation shifts topics. In practice, many production systems use a hybrid: a rolling summary for the bulk of history, plus a separate "critical facts" list that's never summarized away.

def summarize_and_compress(messages, max_tokens=4000):
    """Summarize old messages, keep recent ones verbatim."""
    import tiktoken
    enc = tiktoken.encoding_for_model("gpt-4o")
    total = sum(len(enc.encode(m["content"])) for m in messages)

    if total <= max_tokens:
        return messages  # No need to summarize yet

    # Keep system prompt + last 6 messages verbatim
    system = messages[0]
    recent = messages[-6:]
    old = messages[1:-6]

    if not old:
        return messages  # Nothing to summarize

    # Summarize the old messages
    summary = client.chat.completions.create(
        model="gpt-4o-mini",  # Cheaper model for summarization
        messages=[{
            "role": "system",
            "content": """Summarize this conversation. Preserve:
- User's name, preferences, and stated requirements
- Key decisions made and their reasoning
- Technical context (language, framework, architecture)
- Any constraints or deadlines mentioned
- Safety-critical information (allergies, access levels, etc.)
Be concise but don't lose critical facts."""
        }] + old
    ).choices[0].message.content

    return [
        system,
        {"role": "system",
         "content": f"Previous conversation summary: {summary}"},
        *recent
    ]

SUMMARY_PROMPT = """Summarize the following conversation into these categories:

USER IDENTITY: Name, role, preferences
PROJECT CONTEXT: What they're building, tech stack, architecture
CONSTRAINTS: Deadlines, budgets, requirements, limitations
DECISIONS MADE: What was decided and why
CURRENT STATE: Where the conversation left off
OPEN QUESTIONS: Anything unresolved

Be concise. Use bullet points. Preserve exact numbers, dates, and names."""

# pip install langmem langgraph langchain-openai
from langchain_openai import ChatOpenAI
from langgraph.graph import StateGraph, START, MessagesState
from langgraph.checkpoint.memory import InMemorySaver
from langmem.short_term import SummarizationNode

model = ChatOpenAI(model="gpt-4o")

# Configure automatic summarization
summarization_node = SummarizationNode(
    token_counter=model.get_num_tokens_from_messages,
    model=model.bind(max_tokens=128),
    max_tokens=256,                # max tokens passed to LLM
    max_tokens_before_summary=256, # trigger summarization here
    max_summary_tokens=128,        # max summary size
)

def call_model(state):
    return {"messages": [model.invoke(state["summarized_messages"])]}

# Build graph: summarize → call model
builder = StateGraph(MessagesState)
builder.add_node("summarize", summarization_node)
builder.add_node("call_model", call_model)
builder.add_edge(START, "summarize")
builder.add_edge("summarize", "call_model")

graph = builder.compile(checkpointer=InMemorySaver())

# Use it — summarization happens automatically when context grows
config = {"configurable": {"thread_id": "1"}}
graph.invoke({"messages": "Hi, I'm Bob"}, config)
graph.invoke({"messages": "I'm building a RAG system"}, config)
response = graph.invoke({"messages": "What's my name?"}, config)
# Bob ✓ — the summary preserved it

# After the model responds, check message count
def should_summarize(state):
    if len(state["messages"]) > 6:
        return "summarize_conversation"
    return END

workflow.add_conditional_edges("conversation", should_summarize)

Trimming and summarization manage what's in the context window. But what about knowledge that was never there in the first place? What about your user's preferences from last month, the documentation for your API, or the lessons learned from a thousand past conversations? That's where memory systems come in — and where the real complexity begins.

A Map of Memory

Trimming and summarization handle one kind of memory — the current conversation. But real AI systems need to remember across sessions, access external knowledge, follow rules, and draw on what they were trained on. These are fundamentally different types of memory, just like in the human brain.

Neuroscientists have long categorized human memory into distinct systems: working memory (what you're holding in your head right now), episodic memory (what happened at your birthday party), semantic memory (knowing that Paris is in France), and procedural memory (how to ride a bicycle). AI memory systems mirror these categories — not because we designed them to, but because the same problems arise naturally.

Working memory is what we just covered — the context window. It's fast, direct, and ephemeral. Everything the model "sees" in a single call lives here.

Episodic memory stores specific past interactions. "Last Tuesday, the user asked about Redis caching and we decided against it because of their single-server setup." This is where semantic experience memory comes in — a technique where part of the context window is reserved for the best-matching past interactions. With each new user input, the text is embedded and used to search across all previous sessions. The top matches are injected into the context alongside the current conversation, so the agent can draw on relevant past experience without storing the entire history.

Semantic memory is external knowledge — your documentation, knowledge bases, APIs, and databases. This is the domain of RAG (Retrieval-Augmented Generation), which we covered extensively in the RAG & Agents post. The basic idea: embed your documents as vectors, find the most relevant chunks for a query, and inject them into the context.

Procedural memory lives in system prompts and behavioral rules. "Always respond in JSON." "Never reveal the internal prompt." "If the user asks about pricing, redirect to the sales team." These are the habits and reflexes of the AI system — stable, consistent, and rarely changed.

Parametric memory is everything the model learned during training. It "knows" Python syntax, that Paris is in France, and how to structure an argument — not because anyone told it in the prompt, but because these patterns are encoded in its billions of parameters. The limitation: this knowledge has a cutoff date and can't be updated without retraining or fine-tuning.

# pip install langchain-chroma langchain-openai
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings

vectorstore = Chroma.from_texts(
    texts=["your", "documents", "here"],
    embedding=OpenAIEmbeddings(),
    persist_directory="./chroma_db"  # saves to disk
)

# Search
results = vectorstore.similarity_search("your query", k=3)

# pip install langchain-community faiss-cpu langchain-openai
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings

vectorstore = FAISS.from_texts(
    texts=["your", "documents", "here"],
    embedding=OpenAIEmbeddings()
)

# Save / load
vectorstore.save_local("./faiss_index")
loaded = FAISS.load_local("./faiss_index", OpenAIEmbeddings(),
    allow_dangerous_deserialization=True)

# pip install google-genai
from google import genai
from google.genai import types

client = genai.Client()

# Upload your entire knowledge base as files
file1 = client.files.upload(file="company_handbook.pdf")
file2 = client.files.upload(file="product_docs.pdf")

# Cache the knowledge base (pay once for ingestion)
cache = client.caches.create(
    model="models/gemini-2.5-pro",
    config=types.CreateCachedContentConfig(
        display_name="company_kb",
        system_instruction="Answer based on the knowledge base.",
        contents=[file1, file2],
        ttl="3600s",  # 1 hour cache
    )
)

# Each query now costs 75-90% less for the cached portion
response = client.models.generate_content(
    model="models/gemini-2.5-pro",
    contents="What is our refund policy for enterprise?",
    config=types.GenerateContentConfig(cached_content=cache.name),
)
print(response.text)

Beyond Basic RAG: Three Advances

In the RAG & Agents post, we covered the standard retrieval pipeline: embed your documents, store them in a vector database, find the most similar chunks for a query, and inject them into the prompt. That pipeline treats retrieval as a static, one-shot step that runs before the model generates output.

But what if the agent could decide what, when, and how to retrieve? What if it could build interconnected notes like a researcher? What if retrieval could happen during reasoning, not just before it? These three advances push memory beyond basic RAG.

1. Agentic RAG — The Agent Takes Control

In standard RAG, the vector database is the long-term memory of the LLM — but the LLM has no say in what gets retrieved. It's like a student who gets handed a stack of textbook pages by a librarian and has to make do with whatever was selected. Agentic RAG gives the student direct access to the library card catalog.

In agentic RAG, retrieval becomes a tool that the agent can invoke on its own terms. Instead of a fixed pipeline that always runs the same way, the agent decides:

• Whether to search at all — sometimes the answer is already in the conversation
• Which source to search — documentation? Slack messages? The user's past sessions? A web search?
• What query to use — the agent can reformulate the user's question for better retrieval
• Whether to search again — if the first results aren't sufficient, it can refine and search again

In practice, agentic RAG often uses a router pattern: the agent has access to multiple knowledge sources (documentation, Slack, email, web search, past sessions) as individual tools, and it picks the right one(s) based on the query. It might search your API docs first, find insufficient information, then run a web search for a more recent answer.

This also extends to multi-agent RAG, where specialized retrieval agents each handle a specific knowledge source. A coordinator agent routes the query to the right specialist, collects results, and synthesizes a response. Think of it as having a team of research assistants instead of one generalist.

2. A-MEM — The Zettelkasten for AI

The Zettelkasten method is a note-taking system used by prolific writers and researchers. The core idea: each note contains exactly one unit of knowledge (atomicity), and notes are heavily linked to each other (hypertextual). Over time, you build a web of interconnected ideas where any note can lead you to related concepts you might not have found otherwise.

A-MEM (Agentic Memory) borrows this idea for AI agents. Instead of storing raw conversations or document chunks, each "memory" is an enriched note containing:

• The original interaction (one conversation turn)
• A timestamp
• LLM-generated keywords that capture key concepts
• LLM-generated tags to categorize the interaction
• An LLM-generated contextual description that summarizes the significance

All of this is concatenated and embedded into a single vector. The clever part: when a new memory is added, the system searches for existing memories with similar embeddings. The LLM is then asked to evaluate each candidate and decide which should be linked to the new memory. After linking, connected memories are updated — their tags, keywords, and descriptions evolve to reflect the new relationship.

This creates a living knowledge graph that doesn't just accumulate memories but actively maintains an evolving worldview. When the agent retrieves a memory, it can also traverse links to discover related memories it might not have found through embedding similarity alone.

# pip install langgraph langchain-openai langchain-community faiss-cpu
from langgraph.graph import StateGraph, START, END, MessagesState
from langgraph.prebuilt import ToolNode, tools_condition
from langchain.chat_models import init_chat_model
from langchain_core.messages import HumanMessage
from langchain.tools import tool

model = init_chat_model("gpt-4o", temperature=0)

# Set up your retriever (any LangChain retriever works here)
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings
vectorstore = FAISS.from_texts(["your", "documents", "here"], OpenAIEmbeddings())
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

@tool
def search_knowledge_base(query: str) -> str:
    """Search the knowledge base for relevant information."""
    docs = retriever.invoke(query)
    return "\n\n".join([d.page_content for d in docs])

# Node 1: LLM decides to search or respond directly
def generate_or_search(state: MessagesState):
    response = model.bind_tools([search_knowledge_base]).invoke(state["messages"])
    return {"messages": [response]}

# Node 2: Grade retrieved documents (conditional edge, not a node)
def grade_documents(state: MessagesState):
    question = state["messages"][0].content
    context = state["messages"][-1].content
    grade = model.invoke(f"Answer 'yes' or 'no': Is this context relevant?\nQuestion: {question}\nContext: {context}")
    return "generate_or_search" if "yes" in grade.content.lower() else "rewrite_question"

# Node 3: Rewrite the query for better retrieval
def rewrite_question(state: MessagesState):
    question = state["messages"][0].content
    better = model.invoke(f"Reformulate this question for better search results: {question}")
    return {"messages": [HumanMessage(content=better.content)]}

# Build the graph
workflow = StateGraph(MessagesState)
workflow.add_node(generate_or_search)
workflow.add_node("retrieve", ToolNode([search_knowledge_base]))
workflow.add_node(rewrite_question)

workflow.add_edge(START, "generate_or_search")
workflow.add_conditional_edges("generate_or_search", tools_condition,
    {"tools": "retrieve", END: END})  # END = model answered directly
workflow.add_conditional_edges("retrieve", grade_documents)
workflow.add_edge("rewrite_question", "generate_or_search")  # loop back!

graph = workflow.compile()

3. Search-o1 — RAG During Reasoning

Standard RAG retrieves information before the model starts generating. Agentic RAG lets the agent decide when to retrieve. Search-o1 goes further: it enables retrieval during the reasoning process itself.

Think of it this way. In standard RAG, you hand a student a stack of textbook pages and say "now answer the question." In agentic RAG, the student can ask the librarian for specific books. In Search-o1, the student is working through a problem, realizes mid-thought "I need to check something," looks it up, processes what they found, and continues reasoning. The information arrives exactly when the reasoning process needs it.

Technically, the model is instructed to use special tokens — <|begin_search_query|> and <|end_search_query|> — to trigger a search mid-reasoning. The retrieved documents are then processed by a Reason-in-Documents module: using the search query, the retrieved content, and the current reasoning trace, this module condenses everything into focused reasoning steps that align with the model's thought process.

Why is the Reason-in-Documents step important? Because raw retrieved documents are often long, contain irrelevant sections, and can disrupt the flow of reasoning. By having the model's own reasoning LLM process the retrieved information, the documents are compressed and formatted to fit naturally within the ongoing chain of thought.

Forgetting as a Feature: MemoryBank

Here's a counterintuitive idea: not everything deserves to be remembered. Your brain doesn't retain every conversation you've ever had — it selectively strengthens important memories and lets unimportant ones fade. MemoryBank applies this principle to AI.

Inspired by the Ebbinghaus Forgetting Curve — the psychological model showing that we forget roughly half of what we learn each day unless we actively reinforce it — MemoryBank dynamically adjusts the "strength" of each memory based on usage.

When a memory is retrieved and used during a conversation, its strength is reinforced — making it persist longer in the system. But if a memory hasn't been accessed for a while, it gradually loses strength and may eventually be removed entirely. This is the AI equivalent of spaced repetition — the same technique students use to prepare for exams.

MemoryBank stores three types of information for each user: raw conversation turns (embedded for retrieval), LLM-generated summaries of past events, and a dynamically updated "user portrait" that captures personality traits and preferences. The user portrait is always included in the context; the turns and summaries are retrieved on demand.

This approach aligns with a powerful insight from the mem0 research: only about 10% of information in a conversation deserves permanent storage. The user saying "sounds good" or "let me think about it" doesn't need to be immortalized. But "I'm allergic to penicillin" does. MemoryBank's forgetting mechanism naturally separates the signal from the noise — important information gets retrieved and reinforced, while trivial exchanges fade away.

Don't Forget Keyword Search

With all this talk about embeddings and semantic search, it's tempting to think keyword-based search is obsolete. It isn't. In fact, for certain types of memory retrieval, BM25 keyword search outperforms semantic search.

Consider this: a user asks "What was the configuration for server PROD-DB-07?" Semantic search looks for meaning — it might find documents about database configurations in general. But the user needs the exact string "PROD-DB-07." Keyword search with an inverted index finds it instantly because it's matching exact terms, not meanings.

BM25 (Best Matching 25) is the scoring function behind most keyword search systems. It ranks results by three factors: how often the search term appears in the document (term frequency), how rare the term is across all documents (inverse document frequency), and document length normalization. This makes it excellent for retrieving specific identifiers, error codes, configuration values, and proper nouns — exactly the things semantic search often misses.

In practice, the best memory systems combine both. Semantic search finds conceptually related information; keyword search finds exact matches. This is the hybrid search approach we covered in the RAG & Agents post, and it's just as important for memory retrieval as it is for document retrieval.

# pip install rank_bm25
from rank_bm25 import BM25Okapi
from typing import List

# Your stored memories / conversation turns
memories: List[str] = [
    "User deployed PROD-DB-07 with 16GB RAM and PostgreSQL 15",
    "Discussed Redis caching strategy for the product catalog",
    "User prefers FastAPI over Flask for new microservices",
    "PROD-DB-07 experienced connection timeout at 2:30 AM",
    "Decided to use JWT tokens for API authentication",
]

# Tokenize and build the BM25 index
tokenized = [doc.split() for doc in memories]
bm25 = BM25Okapi(tokenized)

# Retrieve relevant memories for a query
query = "PROD-DB-07 configuration"
scores = bm25.get_scores(query.split())
top_indices = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[:3]

for i in top_indices:
    print(f"  Score {scores[i]:.2f}: {memories[i]}")

# pip install langchain langchain-community langchain-openai rank_bm25 faiss-cpu
from langchain_community.retrievers import BM25Retriever
from langchain.retrievers import EnsembleRetriever
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings

# Your documents (same docs go to both retrievers)
docs = [...]  # list of Document objects

# BM25 retriever (keyword matching)
bm25_retriever = BM25Retriever.from_documents(docs, k=3)

# FAISS retriever (semantic matching)
vectorstore = FAISS.from_documents(docs, OpenAIEmbeddings())
semantic_retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

# Combine with weighted RRF
hybrid_retriever = EnsembleRetriever(
    retrievers=[bm25_retriever, semantic_retriever],
    weights=[0.4, 0.6],  # 40% keyword, 60% semantic
)

results = hybrid_retriever.invoke("PROD-DB-07 connection issues")

Two More Retrieval Techniques Worth Knowing

Before we move on to failure patterns, there are two more memory retrieval approaches that solve common production problems. Both have been around longer than "agentic RAG" became a buzzword, and both remain surprisingly useful.

Semantic Experience Memory — Remembering Past Sessions

Here's a frustration every user has felt: you had a detailed conversation with an AI assistant last week about your project architecture. Today you come back, and it starts from scratch. Semantic experience memory solves this by reserving part of the context window for search results from past interactions.

The mechanism: with each user input, the text is embedded and used as a query against all previous interactions stored in a memory vector store. Part of the context window is reserved for the best matches. The rest of the space goes to the system message, latest input, and most recent turns. This means the agent can recall "you mentioned last Tuesday that you're allergic to shellfish" without the entire previous conversation being in context.

This is different from generic RAG (which retrieves from external documents) and different from conversation history (which only holds the current session). Semantic experience memory specifically searches the agent's own past interactions — making it a form of episodic memory that works across sessions.

Note-Taking — Reading Before Answering

Here's a technique borrowed from how humans handle dense material: before answering a question, first annotate the context. When you're reading a dense research paper and someone asks you a question about it, you don't just skim and answer. You take notes in the margins first, then use those notes to form your answer.

With the self-note approach, the model generates notes on multiple parts of the context before the question is presented. These annotations are then interleaved with the original context when attempting to answer. Research shows good results on multiple reasoning and evaluation tasks — the model essentially "reads the document twice" rather than trying to comprehend and answer simultaneously.

This is especially valuable for long context windows where the model might miss critical details buried in the middle. The note-taking pass forces the model to attend to the entire context, and the resulting notes create an additional "index" that the answer-generation pass can reference.

# Pass 1: Generate margin notes on the context
note_prompt = """Read the following context carefully.
For each important fact, relationship, or detail, write a brief
margin note summarizing it. Number your notes.

CONTEXT:
{long_context}

MARGIN NOTES:"""

notes = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": note_prompt}]
).choices[0].message.content

# Pass 2: Answer the question with context + notes
answer_prompt = """Using the original context AND your notes,
answer the following question.

CONTEXT:
{long_context}

YOUR NOTES:
{notes}

QUESTION: {question}"""

answer = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": answer_prompt}]
).choices[0].message.content

You now know how to store and retrieve memories — from basic trimming to Zettelkasten-inspired networks to mid-reasoning retrieval and note-taking. But even the best memory systems fail in predictable ways. Understanding these failures leads to a bigger insight about how to think about the entire information flow.

Three Fundamental Failure Patterns

Before we catalog everything that can go wrong, let's start with the three failure patterns that every production memory system eventually encounters. These are the ones that show up in your support tickets, your error logs, and your 2 AM Slack alerts.

Failure 1: Stale Memory

Your company updated its parental leave policy in January. A user asks your HR bot about it in March. The bot confidently cites the old policy from 2023 because that document's embedding is semantically identical to the new one, and nobody re-indexed.

Stale memory happens because embedding similarity doesn't consider recency. A 2019 document about "parental leave policy" and a 2024 document about the same topic look nearly identical in vector space. Without explicit recency signals, the retrieval system has no way to prefer the newer version.

Failure 2: Leaked Context

User A tells your bot about their secret project codenamed "Phoenix." User B asks "what projects exist?" User B's response mentions Phoenix. You've just leaked one user's data to another.

Leaked context happens because memory storage isn't namespaced. All users' data lives in the same vector space, and semantic search finds similar content regardless of who wrote it. The retrieval system matched User B's query to User A's data because they were semantically similar.

Failure 3: Polluted Retrieval

A developer asks your coding assistant "How do I deploy to production?" The retrieval system returns chunks about production deployments, production line manufacturing, product management meetings, and productive work habits. The model, now confused by four different meanings of "production," gives a vague, contradictory answer.

Polluted retrieval happens because embedding similarity is imprecise. "Deploy to production" is semantically close to "production line," "product management," and "productive meetings." With a high top-K (retrieving many chunks), noise overwhelms the signal.

Eight Deeper Failure Modes

The three patterns above are the most common, but the Letta Leaderboard — a benchmark designed specifically to test AI memory systems — identified a broader taxonomy of memory failures. These extend beyond the basics and reveal how memory systems break at scale.

4. Unnecessary Searches: The agent searches for information that's already in the context window. A user says "My name is Sarah, I work at Acme Corp." Two messages later, the agent queries the memory system for the user's name. This wastes latency, costs money, and sometimes retrieves conflicting information that overwrites what was already known.

5. Hierarchy Breakdown: Trivial information occupies prime memory while critical facts are buried. The system stores "user likes dark mode" in the same tier as "user is allergic to penicillin." When the context window gets tight, there's no mechanism to prioritize the allergy over the UI preference.

6. Missed Information: The right information is in the context but the model doesn't use it. This often happens with long contexts where important facts are buried in the middle of many paragraphs. The information was successfully retrieved and placed in the prompt, but the model's attention mechanism failed to weight it properly.

7. Silent Overwrites: New information replaces old information without versioning. Sarah's job title changes from "Marketing Manager" to "VP of Marketing." The memory system updates the fact — but now there's no record that she was ever a Marketing Manager. When someone asks "who managed marketing in Q1?" the system has no answer.

8. Isolated Silos: Related facts exist in the memory system but are never connected. The system knows "Server X went down Tuesday" and "Network switch Y was replaced Monday" but never links these events. A human would immediately see the correlation; the flat memory system treats them as unrelated facts.

9. Temporal Confusion: The system can't distinguish between "what happened first" and "what happened recently." It might tell you Sarah's current job title when you ask about her role three years ago, or confuse the order of events in an incident timeline. Without explicit temporal metadata, all facts exist in an eternal present.

10. Scale Collapse: The system works beautifully with 100 stored facts but degrades catastrophically at 10,000. Retrieval precision drops, latency spikes, and the signal-to-noise ratio becomes unmanageable. This is often invisible during development and testing, only revealing itself in production.

The "Lost in the Middle" Problem

Here's something deeply counterintuitive: information placed in the middle of a long context is more likely to be ignored than information at the beginning or end.

The "Lost in the Middle" paper by Liu et al. demonstrated this empirically. They placed a target fact at different positions in a long context and measured how often the model used it correctly. The results followed a U-shaped curve: high accuracy when the fact was near the beginning, high accuracy near the end, and a significant drop in the middle.

This is the serial position effect — a phenomenon well-documented in human psychology. We remember the first things we encounter (primacy effect) and the last things (recency effect), but the middle blurs together. LLMs, despite being fundamentally different from human brains, exhibit the same pattern because of how their attention mechanisms work.

The practical implication is massive: where you place information in the context matters as much as what information you include. If you're building a RAG system, the most critical retrieved chunks should be placed at the very beginning or very end of the retrieved context — never buried in the middle.

Context Rot — When More Is Less

Closely related to "lost in the middle" is a broader phenomenon: context rot. Research consistently shows that model performance degrades as context length increases — even when the context window isn't anywhere near its maximum capacity.

The logic seems simple: more context should mean more information, which should mean better answers. But the empirical data tells a different story.

As you add more tokens to the context, three things happen:

1. Attention dilution: The model's attention mechanism has to spread across more tokens, reducing the weight given to any individual piece of information
2. Noise accumulation: More context means more opportunities for irrelevant information to interfere with the relevant signal
3. Reasoning degradation: Multi-step reasoning becomes harder when the model has to track relationships across a longer sequence

This is why "just stuff everything into a million-token context" is a recipe for failure. The Needle In A Haystack (NIAH) benchmark — the most common test for long-context models — merely tests retrieval: can the model find a specific fact hidden in long text? Most modern models ace this test. But retrieval is the easy part. The RULER benchmark demonstrated that models performing well on NIAH showed significant performance drops when tested on multi-hop tracing, aggregation, and counting — tasks that require actual reasoning over the context, not just finding a needle.

The conclusion is clear: context quality matters more than context quantity. This is the foundation of the paradigm shift we'll explore in Part 4.

These failures aren't random. They all stem from the same root cause: treating the context window as a dumping ground rather than as a carefully engineered input. There's a better way to think about it — and it's called context engineering.

The LLM as a Function

Let's step back and think about what an LLM really is. Strip away the chat interfaces, the API wrappers, the prompt templates. At its core, an LLM is a function: it takes a sequence of input tokens, processes them through billions of parameters, and outputs a sequence of tokens.

That's it. The entire "intelligence" of the system comes from two things: the quality of the model (its parameters) and the quality of the input (the context). You can improve the model through training or fine-tuning — expensive, slow, and requires expertise. Or you can improve the input — the context you send to the model.

Context engineering is the art and science of optimizing those input tokens so they produce the best possible output for a given task. It's not about filling the context window. It's about strategically choosing and placing information so the model can do its best work.

Think of it like writing a brief for a brilliant but literal-minded consultant. This consultant will do exactly what you tell them, using exactly the information you provide.

Five Principles of Context Engineering

Based on research from Anthropic, the RULER benchmark findings, and production experience with large-scale AI systems, five principles emerge for engineering effective context.

Principle 1: Relevance — Every Token Earns Its Place

The most common mistake in context engineering is including too much. Every token you add to the context has a cost: financial (you pay per token), computational (the model processes every token), and attentional (each token dilutes the attention available for other tokens).

Relevance means asking a simple question for every piece of information: "Does this serve the current task?" If the user is asking about database configuration, their chat preferences from last month probably don't need to be in the context. MemoryBank's forgetting mechanism is one implementation of this principle — it naturally surfaces frequently-used (relevant) information and lets rarely-used information fade.

Principle 2: Diversity — Avoid Redundancy

Retrieving five chunks that all say "use connection pooling for database performance" wastes four context slots that could contain new information. Maximal Marginal Relevance (MMR) is the key technique here: it balances relevance to the query against redundancy with already-selected chunks.

# pip install langchain-chroma langchain-openai
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings

# Build your vector store (works with FAISS, Pinecone, Qdrant too)
vectorstore = Chroma.from_texts(
    texts=your_documents,
    embedding=OpenAIEmbeddings()
)

# One line: switch from similarity to MMR
retriever = vectorstore.as_retriever(
    search_type="mmr",
    search_kwargs={
        "k": 5,            # return 5 documents
        "fetch_k": 50,     # consider 50 candidates for diversity
        "lambda_mult": 0.3  # 0.0 = max diversity, 1.0 = max relevance
    }
)

docs = retriever.invoke("your query here")

Principle 3: Ordering — Position Is Power

We covered the "lost in the middle" phenomenon in Part 3. The practical takeaway: put the most critical information at the beginning and end of the context. System prompts naturally go first. The user's latest message naturally goes last. Everything in between should be ordered by importance, with the most critical items nearest to these anchors.

For RAG systems, this means the highest-ranked retrieved chunks should be placed either immediately after the system prompt or immediately before the user's message — not sandwiched in the middle of a long conversation history.

Principle 4: Freshness — Recency-Weighted Scoring

Stale memory (Failure 1) happens because retrieval systems treat all information as equally timely. The fix: blend recency into your scoring function.

A simple approach: final_score = α · semantic_similarity + (1 - α) · recency_score, where recency decays exponentially from the document's last-updated timestamp. For most applications, α = 0.7 (favoring semantic relevance) is a good starting point, with adjustments based on how frequently your domain's information changes.

Principle 5: Specification — Context as Code

Here's the paradigm shift: treat your context like you treat your code. The context window isn't just an input — it's a specification for the model's behavior. Just as you wouldn't deploy code without testing, you shouldn't deploy a context configuration without testing.

This means: version your system prompts. Track what context you're sending. Monitor retrieval quality. Set up automated tests that verify the right information surfaces for known queries. Create regression tests for the failure patterns from Part 3. Context engineering is an engineering discipline, not a creative exercise.

This insight transforms how you think about AI operations. The context you send isn't just an input — it's a persistent artifact that explains the agent's decisions. Sophisticated agents already use this pattern: they maintain a PLAN.md file that persists across context cropping. When the orchestrator trims old messages, the plan survives, providing continuity and an audit trail of what the agent intended to do and why.

What to Track: The Context Taxonomy

Before you can optimize context, you need to know what types of context exist. Most developers only think about the conversation history. In reality, the full context landscape includes four categories:

In practice, not everything is useful and many other things should be tracked. The key is deciding beforehand what to capture. This taxonomy gives you a starting framework — especially useful when debugging why an agent took an unexpected action. Was the issue in what tools it used (agent behavior), what the user asked for (user behavior), what knowledge was retrieved (knowledge sources), or how the system was configured (system-level)?

Multi-Agent Context

In systems with multiple AI agents working together — an orchestrator routing tasks to specialists — context engineering becomes even more critical. Each agent needs to see different information, and sharing everything between all agents creates unnecessary cost and confusion.

The orchestrator sees the big picture: the user's query, the task plan, and summaries from each specialist. But each specialist sees only what it needs. The research agent gets the search query and retrieved documents. The coding agent gets the code context and requirements. The summary agent gets the draft and user preferences. No agent is overwhelmed with irrelevant context from other agents' work.

This is context engineering at the architectural level — designing not just what goes into a single context window, but how context flows across an entire multi-agent system.

SYSTEM_PROMPT = """
You are a helpful research assistant.

<INSTRUCTIONS>
* Create a plan for the user's query
* If you lack information, use tools to search
* Prioritize clarity and use the current date for SOTA
</INSTRUCTIONS>

<TOOLS>
{information_about_tools}
</TOOLS>

<DATE>
{current_date}
</DATE>

<USER_QUERY>
{user_query}
</USER_QUERY>
"""

# Step 1: Create a plan
messages = [{"role": "system", "content": SYSTEM_PROMPT}]

# Step 2: User provides feedback on the plan
messages = [
  {"role": "system", "content": SYSTEM_PROMPT},
  {"role": "assistant", "content": "Here is my plan..."},
  {"role": "user", "content": "Search for surveys instead."}
]

# Step 3: CROP — only keep the updated plan
messages = [
  {"role": "system", "content": SYSTEM_PROMPT},
  {"role": "assistant", "content": PLAN_MD},  # Persistent artifact
]

# Step 4: Tool call for search
messages = [
  {"role": "system", "content": SYSTEM_PROMPT},
  {"role": "assistant", "content": PLAN_MD},
  {"role": "assistant", "content": "<think>Search on ArXiv</think>",
   "tool_call": {"name": "search_arxiv", ...}},
  {"role": "tool", "content": ABSTRACTS},
]

# Step 5: Sub-agent gets MINIMAL context
messages_summary_agent = [
  {"role": "system", "content": "Summarize these papers."},
  {"role": "user", "content": PAPERS},
]

# Step 6: Final synthesis — CROPPED again
messages = [
  {"role": "system", "content": SYSTEM_PROMPT},
  {"role": "assistant", "content": UPDATED_PLAN},
  {"role": "assistant", "content": "Summaries: ..."},
]

Context as Specification

Let's close with the biggest mental shift in context engineering: the context window IS the product.

When you're building a traditional application, you write code that processes inputs and produces outputs. The code is the product. When you're building an AI application, the LLM processes context and produces responses. The context — how you assemble it, what you include, how you order it, how you test it — is the product.

This is why the techniques we've covered in this post matter so much. Memory management (Part 1) determines what historical information is available. Memory architecture (Part 2) determines how efficiently information can be stored and retrieved. Failure awareness (Part 3) tells you what can go wrong. And context engineering (this section) provides the framework for putting it all together.

The systems that get context engineering right — that treat their context window as a first-class engineering artifact — consistently outperform systems with bigger models, more parameters, and larger context windows. The context is the specification. Engineer it accordingly.

This post covered memory within a single context window and across sessions. But what happens when flat memory can't represent the relationships between facts? When you need to know not just what happened but when, why, and how it connects to other events? That's the domain of graph memory systems — the subject of the next post in this series.