How LLMs Learn to Find Answers and Take Action

Let's walk through a concrete example. Suppose you have 10,000 documents and you search for "transformer architecture."

For the word "transformer":

• It appears 15 times in Document A → TF = 15
• 200 out of 10,000 documents contain it → IDF = log(10,000 / 200) = 3.91
• TF-IDF for "transformer" in Doc A = 15 × 3.91 = 58.7

For the word "architecture":

• It appears 8 times in Document A → TF = 8
• 1,500 out of 10,000 documents contain it → IDF = log(10,000 / 1,500) = 1.90
• TF-IDF for "architecture" in Doc A = 8 × 1.90 = 15.2

Document A's total score = 58.7 + 15.2 = 73.9

Notice how "transformer" contributes much more to the score than "architecture" — because it's a rarer, more informative word. This is exactly the behavior we want.

How does the data get organized for fast retrieval? Through a structure called an inverted index. Instead of storing "Document → list of words," it stores "Word → list of documents." Think of it as the index at the back of a textbook: you look up a term and it tells you which pages mention it.

A production evolution of TF-IDF is BM25 (Best Matching 25) — it adds document length normalization so that longer documents don't automatically score higher. BM25 and its variants are still the backbone of systems like Elasticsearch (built on Lucene's inverted index) and remain formidable baselines that surprisingly often beat more sophisticated approaches.

One subtlety worth noting: before any scoring happens, the query must be broken into individual terms — a process called tokenization. The simplest method is splitting by words, but this can destroy multi-word meanings ("hot dog" becomes "hot" and "dog," neither retaining the original meaning). Production systems handle this by treating common n-grams as terms, converting to lowercase, removing punctuation, and eliminating stop words like "the" and "is." Classical NLP packages like NLTK, spaCy, and CoreNLP provide these tokenization features.

The naive approach to finding the closest vectors is simple: compare the query vector against every single vector in the database. This guarantees perfect results but is painfully slow. For a database of 100 million vectors, each with 1,024 dimensions? That's 100 billion floating-point operations per query.

Production systems use approximate nearest neighbor (ANN) algorithms that trade a tiny bit of accuracy for massive speed gains. Here are the major approaches:

The key trade-off is between indexing (build time + memory) and querying (speed + accuracy). HNSW gives the best query accuracy but needs significant memory and build time. LSH is cheap to build but less accurate. IVF + Product Quantization offers a good balance for very large datasets. Annoy is lightweight and uses static files, great for read-heavy workloads.

For benchmarking, three resources are essential: the ANN-Benchmarks website compares vector search algorithms across datasets on recall, queries per second, build time, and index size. BEIR (Benchmarking IR) evaluates retrieval systems across 14 common retrieval benchmarks. And the MTEB (Massive Text Embedding Benchmark) evaluates embeddings across retrieval, classification, and clustering tasks — it's the go-to leaderboard for choosing an embedding model.

Dense retrieval (embedding-based search) is powerful, but has specific failure modes you should know about:

No relevant results exist, but you get results anyway. If you ask "What is the mass of the moon?" against a movie database, the system still returns the nearest vectors — even though none are relevant. One heuristic is to set a maximum distance threshold for relevance. Any result beyond that threshold is treated as "no match found." In practice, many search systems present the best results and let the user decide, tracking whether users actually click on them to improve future versions.

Exact phrase matching fails. When a user wants to find an exact match for a specific phrase, embeddings can fuzz over the exact words. This is why hybrid search (combining semantic with keyword search) is recommended over relying solely on dense retrieval.

Domain transfer problems. Dense retrieval models trained on internet and Wikipedia data often perform poorly on specialized domains like legal, medical, or financial text — without sufficient domain-specific training data. The embedding model doesn't understand your domain's jargon.

Multi-sentence answers get lost. If the answer to a question spans multiple sentences or chunks, individual chunks may not capture enough context for a good embedding match. This is where chunking strategy becomes critical — more on that in Part 3.

One useful rule of thumb: Anthropic suggests that if your knowledge base is smaller than 200,000 tokens (about 500 pages), you can include the entire knowledge base directly in the prompt — no RAG needed. It would be great if other model providers gave similar guidance for when RAG is and isn't necessary.

Just like classification models, embedding models can be fine-tuned to work better for your specific search task. The key insight: a query and its best result aren't always "similar" in the general sense — they just need to be close in embedding space.

The fine-tuning process uses contrastive learning. For each document in your dataset, you create training pairs:

• Positive pairs: Queries that the document correctly answers (e.g., "Interstellar release date" paired with "Interstellar premiered on October 26, 2014, in Los Angeles.")
• Negative pairs: Queries that the document does NOT answer (e.g., "Interstellar cast" paired with the same premiere sentence)

Training pushes relevant query-document pairs closer together in embedding space and irrelevant pairs further apart. Before fine-tuning, all queries about "Interstellar" might be equally close to the document. After fine-tuning, only the truly relevant queries are close.

The Sentence Transformers library is the standard tool for this. It supports training with various loss functions, including contrastive loss, and integrates with popular model hubs. See the "Retrieve & Re-Rank" section of their documentation for step-by-step instructions.

Here's a minimal but complete example of building a dense retrieval system with search and reranking. We'll use Cohere for embeddings and reranking, and FAISS for the vector index.

Step 1: Embed your documents

import cohere
import numpy as np
import faiss

co = cohere.Client('your-api-key')

# Your documents (chunks of text)
texts = [
    "The transformer architecture uses self-attention...",
    "Performance optimization techniques for web apps...",
    "Warfarin interacts with vitamin K...",
    # ... more documents
]

# Embed all documents
response = co.embed(
    texts=texts,
    input_type="search_document"
).embeddings
embeds = np.array(response, dtype='float32')
# Result: (num_docs, 4096) matrix

Step 2: Build the FAISS index

# Create a flat L2 index (exact search)
dim = embeds.shape[1]  # 4096
index = faiss.IndexFlatL2(dim)
index.add(embeds)
print(f"Index contains {index.ntotal} vectors")

Step 3: Search

def search(query, k=10):
    # Embed the query
    query_embed = co.embed(
        texts=[query],
        input_type="search_query"
    ).embeddings
    query_embed = np.array(query_embed, dtype='float32')

    # Find k nearest neighbors
    distances, indices = index.search(query_embed, k)
    return [(texts[i], distances[0][j])
            for j, i in enumerate(indices[0])]

results = search("drug interactions with warfarin")
for text, dist in results[:3]:
    print(f"  {dist:.1f}  {text[:80]}...")

Step 4: Rerank for precision

# Take top 10 from search, rerank to find best 3
query = "drug interactions with warfarin"
search_results = search(query, k=10)
docs = [text for text, _ in search_results]

reranked = co.rerank(
    query=query,
    documents=docs,
    top_n=3,
    return_documents=True
)

for result in reranked.results:
    print(f"  {result.relevance_score:.3f}  "
          f"{result.document.text[:80]}...")

For production: Replace IndexFlatL2 with IndexIVFPQ for datasets over 100K vectors. Use a managed vector database (Pinecone, Weaviate, or Qdrant) if you need to add/remove vectors without rebuilding the index.

Let's see how keyword search (BM25) compares to the dense retrieval system we built above — using the same query on the same documents.

Set up BM25 keyword search:

from rank_bm25 import BM25Okapi
from sklearn.feature_extraction import _stop_words
import string

def bm25_tokenizer(text):
    tokenized_doc = []
    for token in text.lower().split():
        token = token.strip(string.punctuation)
        if len(token) > 0 and token not in \
            _stop_words.ENGLISH_STOP_WORDS:
            tokenized_doc.append(token)
    return tokenized_doc

# Tokenize and index documents
tokenized_corpus = [bm25_tokenizer(t) for t in texts]
bm25 = BM25Okapi(tokenized_corpus)

def keyword_search(query, top_k=3):
    bm25_scores = bm25.get_scores(bm25_tokenizer(query))
    top_n = np.argpartition(bm25_scores, -top_k)[-top_k:]
    hits = sorted(
        [{'id': idx, 'score': bm25_scores[idx]}
         for idx in top_n],
        key=lambda x: x['score'], reverse=True
    )
    for hit in hits[:top_k]:
        print(f"  {hit['score']:.3f}  "
              f"{texts[hit['id']][:80]}...")
    return hits

Compare the same query:

query = "how precise was the science"

print("=== Dense Retrieval (Semantic) ===")
search(query)  # From our FAISS search above

print("\n=== BM25 (Keyword) ===")
keyword_search(query)

# Dense retrieval correctly finds:
#   "received praise for scientific accuracy..."
# BM25 matches on the word "science" instead:
#   "science fiction film co-written by Nolan..."

The dense retrieval system finds the document about scientific accuracy — the semantically correct answer — even though it doesn't contain the word "precise." BM25 matches on the keyword "science" and returns the wrong document. This is exactly why semantic search exists.

But BM25 wins for exact terms. Search for a specific name like "Kip Thorne" and BM25 nails it instantly while semantic search might return vaguely related physics documents. This is why production systems combine both approaches.

You can build a complete RAG system without any API calls — running everything locally on your machine. Here's how using a quantized model (Phi-3), a local embedding model, and FAISS.

Step 1: Load a local generation model

# Download a quantized GGUF model
# !wget https://huggingface.co/microsoft/
#   Phi-3-mini-4k-instruct-gguf/resolve/main/
#   Phi-3-mini-4k-instruct-fp16.gguf

from langchain import LlamaCpp

llm = LlamaCpp(
    model_path="Phi-3-mini-4k-instruct-fp16.gguf",
    n_gpu_layers=-1,
    max_tokens=500,
    n_ctx=2048,
    seed=42,
    verbose=False
)

Step 2: Load a local embedding model

from langchain.embeddings.huggingface import (
    HuggingFaceEmbeddings
)

# A small, high-quality embedding model
embedding_model = HuggingFaceEmbeddings(
    model_name='thenlper/gte-small'
)

Step 3: Build the vector database

from langchain.vectorstores import FAISS

# Create a local FAISS index from your documents
db = FAISS.from_texts(texts, embedding_model)

Step 4: Create the RAG pipeline

from langchain import PromptTemplate
from langchain.chains import RetrievalQA

template = """<|user|>
Relevant information:
{context}

Answer concisely using the information above:
{question}<|end|>
<|assistant|>"""

prompt = PromptTemplate(
    template=template,
    input_variables=["context", "question"]
)

rag = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type='stuff',
    retriever=db.as_retriever(),
    chain_type_kwargs={"prompt": prompt}
)

# Ask a question!
rag.invoke('What was the total revenue?')

The local model won't match the quality of larger managed models (no inline citations, for instance), but this gives you a complete, private RAG system running entirely on your hardware — no API costs, no data leaving your machine.

Hypothetical Document Embeddings (HyDE) is one of the most creative query rewriting techniques. Instead of searching with the question, you search with a fake answer.

Here's the idea: ask the LLM to generate a hypothetical document that would answer the question — even though it might be completely wrong. Then use that document as the search query instead of the original question.

Why would searching with a wrong answer work?

Because the vocabulary mismatch between a question and its answer is often huge. Consider:

• Question: "Which politicians complained about the budget not being balanced?"
• Actual document: 'Senator Paxton: "I can't stand the sight of our enormous deficit!"'

The question uses words like "complained" and "balanced." The actual document uses "can't stand" and "deficit." There's barely any word overlap. But a hypothetical answer generated by the LLM might say:

"Senator Mark Wellington stated, 'This government's failure to balance the budget is unacceptable.' MP Emily Fraser remarked, 'We cannot continue this path of reckless spending without addressing the deficit.'"

This hypothetical answer is completely made up — those politicians don't exist. But it uses the same kind of language that real politicians use. Its embedding will be much closer to the actual Senator Paxton quote than the original question's embedding would be.

The key insight: A fake answer is closer in embedding space to real answers than the question itself is. The factual accuracy doesn't matter — the linguistic style is what makes the retrieval work.

Practical tips: You can use a smaller, cheaper LLM for HyDE since factual accuracy doesn't matter. Keep the generated document roughly the same length as your chunks. Watch for topic drift — the generated text may wander off-topic after the first few sentences.

HyDE is just one of several query expansion techniques. Here are the others you should know about:

Pseudo-Relevance Feedback (PRF)

PRF is a tried-and-tested technique from the information retrieval field. The idea: use your original query to retrieve a first batch of documents, then extract salient terms from those documents and add them back to your query. For example, searching "Which politicians complained about the budget not being balanced?" might return documents containing terms like "fiscal policy," "deficit," "financial mismanagement," and "budgetary reforms." Adding these terms to the original query dramatically improves recall on the second search pass. PRF is simple, effective, and doesn't require an LLM — but it's susceptible to topic drift if the initial results are off-target.

Query2Doc

Similar to HyDE, Query2Doc uses an LLM to generate a hypothetical document, but the key difference is how the generated text is used. Rather than replacing the query entirely, Query2Doc combines the generated document with the original query to create a richer search input.

Query2Doc2Keyword

This technique (Li et al.) chains three steps: generate a hypothetical document from the query, extract salient keywords from that document, then add those keywords to the original query. The authors improve keyword quality using the self-consistency method — repeating the keyword generation multiple times and selecting only keywords that appear across multiple generations. This filters out noise and keeps only the most reliable expansion terms.

Document-side rewriting

All the techniques above modify the query to bring it closer to the document space. An alternative approach flips this: modify the documents to bring them closer to the query space. Techniques like doc2query (generate potential questions each document could answer) and contextual retrieval (add context headers to chunks) do exactly this. The upfront cost is higher — you process every document once — but it eliminates query-rewriting latency at inference time.

Query Routing

Once you have multiple data sources, the LLM can learn to route queries to the right one. For example: HR questions go to Notion, customer data goes to Salesforce, financial reports go to the document store. The model examines the query and decides which data source(s) to search — or whether multiple sources are needed.

The road to Agentic RAG

Notice the progression: simple query rewriting → multi-query → multi-hop → query routing. Each step delegates more responsibility to the LLM. At the end of this progression, the LLM isn't just rewriting queries — it's deciding what to search, where to search, and when to stop. The data sources become tools, and the LLM becomes an agent. This is exactly the transition we'll explore in Part 4.

Beyond the standard retrieve-then-rerank pattern, researchers have developed alternative retrieval paradigms that rethink how documents are found:

When the refine stage summarizes retrieved documents, there are two fundamentally different approaches:

Tightly-coupled summarizers

Both extractive and abstractive summarizers can be tightly coupled with the generator LLM — meaning they're trained using feedback from the LLM itself (Xu et al.).

For extractive summarizers: each sentence is prefixed to the input query, and the likelihood of the LLM generating the correct output is measured. Sentences with high likelihood become positive training examples; sentences below a threshold become negatives. The model is trained with contrastive learning on these triplets.

For abstractive summarizers: a larger LLM generates candidate summaries using prompt templates. Each summary is evaluated by measuring output token likelihood when it's used as context. The best summary (highest likelihood) becomes the training target. During inference, if no summary improves the LLM's output likelihood compared to no summary at all, an empty summary is returned — effectively filtering out irrelevant documents.

Practical note: Tightly-coupled summarizers are expensive to train initially but effective at removing noise. If you change your target LLM, you may need to retrain the summarizers — there's a slight performance degradation when transferring across models.

Here are four advanced techniques that push RAG beyond the basic pipeline:

Cross-encoders and ColBERT are encoder-based models. But decoder LLMs can also be trained to rank documents directly. There are three approaches:

Training these rankers via distillation: Models like RankGPT, RankVicuna, and RankZephyr are trained by distilling from larger LLMs. The process for RankVicuna: (1) feed training queries through BM25 to get candidate documents, (2) pass these to a large LLM which generates a rank-ordered list, (3) use these query + ranked-list pairs to fine-tune a smaller LLM.

The RankVicuna authors found two interesting results: as the quality of the first-level retrieval increases, the gains from reranking decrease (diminishing returns). And augmenting training data by shuffling the input order of candidate documents improved model performance — it prevents the model from learning positional biases.

Practical tip: You can combine retrieval scores with reranking scores for the final relevance ranking. This is useful when you need to enforce keyword weighting or factor in metadata like publication date (prefer newer documents).

Evaluating a search system requires three things: a text archive, a set of test queries, and relevance judgments — human labels indicating which documents are relevant for each query.

Average Precision (AP) for a single query:

Imagine your search system returns 5 results for a query, and results at positions 1, 3, and 5 are relevant (positions 2 and 4 are irrelevant). The calculation focuses only on the relevant positions:

Now compare: if the same 3 relevant documents were at positions 3, 4, and 5 instead:

The first system scores 0.76, the second 0.48 — AP naturally rewards systems that put relevant results higher in the ranking.

Mean Average Precision (MAP) is simply the mean of AP scores across all queries in your test suite. This gives you a single number to compare entire search systems.

Normalized Discounted Cumulative Gain (nDCG)

MAP treats relevance as binary — a document is either relevant or not. But sometimes one document is more relevant than another. nDCG handles this with graded relevance (e.g., 0 = irrelevant, 1 = somewhat relevant, 2 = highly relevant). It also applies a logarithmic discount — relevant documents at lower positions contribute less to the score. nDCG is normalized to [0, 1] by dividing by the ideal ranking's score.

LLM-as-a-Judge: While human evaluation is always preferred, it doesn't scale. The RAGAS library automates evaluation by having a capable LLM (like GPT-4) score your system's outputs across multiple axes — faithfulness, answer relevance, context precision, and more. Human evaluation along four axes (fluency, perceived utility, citation recall, citation precision) from the paper "Evaluating Verifiability in Generative Search Engines" remains the gold standard for production systems.

RAGAS is a library that automates RAG evaluation by using an LLM as a judge. Instead of manually labeling thousands of query-answer pairs, you let a capable model (like GPT-4) evaluate your system's outputs.

What you need:

• A set of test queries
• For each query: the retrieved context, the generated answer, and (ideally) a ground-truth answer

from ragas import evaluate
from ragas.metrics import (
    context_precision,
    context_recall,
    faithfulness,
    answer_relevancy,
)
from datasets import Dataset

# Your test data
data = {
    "question": [
        "What is our parental leave policy?",
        "How do I request time off?",
    ],
    "answer": [
        "Our parental leave policy provides 16 weeks...",
        "To request time off, submit a request in HR Portal...",
    ],
    "contexts": [
        ["Policy doc chunk 1...", "Policy doc chunk 2..."],
        ["HR Portal guide chunk 1...", "Time-off FAQ chunk..."],
    ],
    "ground_truth": [
        "The 2024 policy provides 16 weeks of paid leave.",
        "Employees request time off through the HR Portal.",
    ],
}

dataset = Dataset.from_dict(data)
results = evaluate(
    dataset,
    metrics=[
        context_precision,
        context_recall,
        faithfulness,
        answer_relevancy,
    ],
)
print(results)
# {'context_precision': 0.83, 'context_recall': 0.91,
#  'faithfulness': 0.95, 'answer_relevancy': 0.88}

How to interpret the results:

• Low context precision → Your retriever is returning too many irrelevant documents. Fix: better reranking, metadata filters, or tighter similarity thresholds.
• Low context recall → Your retriever is missing relevant documents. Fix: better chunking, add synonyms, try hybrid search.
• Low faithfulness → The LLM is making claims not supported by the context. Fix: add Chain-of-Note, improve prompt instructions, use a more instruction-following model.
• Low answer relevancy → The answer doesn't address the question. Fix: improve query rewriting, check that the right documents are being retrieved.

Here are the concrete tradeoffs when choosing chunk size:

Practical guidance:

• Always add overlapping between chunks (10-20% of chunk size) to avoid cutting mid-thought
• Chunk size must not exceed your embedding model's context limit
• Consider metadata-aware chunking: use section headers, paragraph boundaries, and document structure to determine chunk boundaries
• For code: use language-specific splitters that respect function/class boundaries
• For Q&A: each question-answer pair can be its own chunk
• Augment chunks with parent document title and summary — this is Anthropic's "contextual retrieval" approach that dramatically improves retrieval quality

There's no universal best size. The only reliable approach: try multiple sizes, measure retrieval quality on your actual queries, and pick the winner.

Beyond the four strategies shown above, here are additional approaches used in production:

Token-based chunking — Instead of splitting by characters or sentences, use the generative model's tokenizer as the unit. Split documents into chunks of N tokens using the same tokenizer your downstream LLM uses (e.g., Llama 3's tokenizer). This makes it easier to control exactly how many tokens each chunk consumes in the context window. The downside: if you switch to a different LLM with a different tokenizer, you'd need to re-index everything.

Sliding window chunking — When chunk boundaries split related content, the neighboring chunk's context is lost. Sliding window fixes this: each piece of text can appear in multiple overlapping chunks, so neighboring context is always preserved in at least one chunk. Think of it as moving a window across the document rather than cutting it into discrete pieces.

Metadata-aware chunking — Use paragraph boundaries, section headers, and subsection structure to determine where to split. If metadata isn't readily available, document parsing tools like Unstructured can extract this information automatically. This preserves the document's logical structure.

Layout-aware chunking — A more sophisticated version of metadata-aware chunking that uses computer vision to extract layout information: text placement, font sizes, titles, subtitles, table boundaries, and image regions. Tools include AWS Textractor, Unstructured, and layout-aware language models like LayoutLMv3. For example, knowing a subsection's scope lets you prepend the subsection title to every chunk within it.

Contextual retrieval — Anthropic's approach: augment each chunk with context from its parent document. An LLM generates a short context (50-100 tokens) explaining the chunk and its relationship to the whole document. This context is prepended to the chunk before indexing. You can also augment chunks with: the questions they could answer, extracted entities (so error code EADDRNOTAVAIL becomes searchable), or product descriptions and reviews for e-commerce data.

Multi-level embeddings — Use multiple levels of granularity: sentence embeddings, paragraph embeddings, section embeddings, and even document-level summary embeddings. You can use different embedding models at each level — more expensive models at higher granularity levels where there are fewer items to embed. During retrieval, you can start from the top level and drill down like a tree, or target a specific level directly.

A warning about document parsing: Effective document parsing — extracting section boundaries, detecting tables, handling heterogeneous formats — is the unglamorous foundation of any RAG system. If your data includes domain-specific text, even sentence tokenization isn't trivial (abbreviations confuse naive period-splitting). NLTK's Punkt tokenizer can be trained unsupervised on your domain text, and spaCy, Stanza, and ClarityNLP offer additional options. A large proportion of RAG failure modes trace back to poor document parsing.

Embeddings are famously opaque — you know that two vectors are close or far, but you can't see why. What features does each dimension encode? Why are two sentences sometimes closer than you'd expect?

Sparse Autoencoders (SAEs) offer a window into this black box. The idea: a language model learns millions of features, but only a few are active for any given input. SAEs decompose the embedding into independent, interpretable features.

When researchers applied SAEs to T5 embeddings, they discovered features like:

• Presence of negation — fires when text contains "not," "never," "can't"
• Expression of possibility — fires on "might," "could," "perhaps"
• Employment and labor concepts — fires on job-related text
• Possessive syntax at sentence start — fires on "My," "Their," "Our"

Features range from lexical (specific words) to syntactic (grammar patterns) to topical to pragmatic (tone, intent). Each feature activates on specific text samples, making it possible to understand — and even steer — what the embedding encodes.

Practical implication: If you know what features your embedding represents, you can perform "semantic edits" — adjusting specific features to change the retrieval behavior without retraining the model.

Espresso Sentence Embeddings (Li et al.) takes a different approach to compression: instead of reducing dimensions, it reduces layers. Embeddings are typically extracted from the last layer of the model, but Espresso trains the lower layers to produce high-quality embeddings too, using KL-divergence loss between the final layer and each lower layer. Removing half the layers gives a 2x speed improvement while preserving 85% of the original performance.

The Sentence Transformers library lets you combine Matryoshka dimension reduction with Espresso layer reduction using Matryoshka2dLoss — compressing both width and depth simultaneously.

Training Matryoshka embeddings:

from sentence_transformers import (
    SentenceTransformer, SentenceTransformerTrainer, losses
)
from datasets import load_dataset

model = SentenceTransformer("all-mpnet-base-v2")
train_dataset = load_dataset("csv", data_files="finetune_dataset.csv")

# Standard contrastive loss
loss = losses.MultipleNegativesRankingLoss(model)

# Wrap with Matryoshka — optimize for multiple truncation points
loss = losses.MatryoshkaLoss(model, loss, [768, 512, 256, 128])

trainer = SentenceTransformerTrainer(
    model=model,
    train_dataset=train_dataset,
    loss=loss,
)
trainer.train()
# Result: truncate to 128 dims (8.3% of original) and
# retain 98.37% of performance

Int8 quantization: Between full float32 and extreme binary (1-bit), there's int8 quantization — converting each dimension to an integer between -127 and 127 (or 0-255 unsigned). This gives 4x compression instead of 32x, but with better precision. A calibration dataset is used to determine min/max values per dimension for the conversion.

from sentence_transformers.quantization import quantize_embeddings

model = SentenceTransformer("all-mpnet-base-v2")
embeddings = model.encode([
    "I heard the horses are excited for Halloween.",
    "Dalmatians are the most patriotic of dogs.",
])

# Binary: 32x compression, uses bitwise operations
binary = quantize_embeddings(embeddings, precision="binary")

# Int8: 4x compression, better precision than binary
int8 = quantize_embeddings(embeddings, precision="int8")

Surprising finding: For some embedding models, binary quantization actually outperforms int8 despite having far less precision. This is largely due to the challenge of mapping float values to int8 buckets using the calibration dataset. When in doubt, benchmark both on your actual queries.

Chroma is the simplest vector database to get started with — it's open-source and runs in-memory locally:

import chromadb

# Create client and collection
chroma_client = chromadb.Client()
collection = chroma_client.create_collection(name="mango_science")

# Add documents with metadata
chunks = [
    "353 varieties of mangoes are now extinct",
    "Mangoes are grown in the tropics"
]
metadata = [
    {"topic": "extinction", "chapter": "2"},
    {"topic": "regions", "chapter": "5"}
]
collection.add(
    documents=chunks,
    metadatas=metadata,
    ids=[str(i) for i in range(len(chunks))]
)

# Query with metadata filtering
results = collection.query(
    query_texts=["Where are mangoes grown?"],
    n_results=2,
    where={"chapter": {"$ne": "2"}},          # metadata filter
    where_document={"$contains": "grown"}      # text filter
)

Notice how Chroma handles embedding generation automatically — you don't need to manage embedding models separately. The where clause works like SQL filtering on metadata, and where_document filters on the actual text content.

Evaluating which vector database to use:

• What retrieval mechanisms does it support? Hybrid search?
• What embedding models and vector search algorithms are supported?
• How scalable is it for both data storage and query traffic?
• How long does indexing take? Can you bulk add/delete?
• What's the query latency for different retrieval algorithms?
• For managed solutions: is pricing based on document volume or query volume?

Here's a powerful way to think about RAG that clarifies when to use it: the LLM's context window is like RAM, and RAG is like virtual memory.

In an operating system, memory is organized in a hierarchy:

• CPU registers — tiny, blazingly fast (the LLM's internal weights)
• RAM — limited but directly accessible (the context window)
• Disk/SSD — massive but slow (your vector database)

When a program needs data that isn't in RAM, the OS swaps it in from disk and swaps out data that isn't currently needed. RAG does exactly the same thing for LLMs — it swaps relevant information into the context window on demand.

This analogy reveals three key insights:

1. Just like RAM, the context window has a fixed size. You need to be strategic about what goes in. Stuffing everything in (the "long context" approach) is like buying a computer with 1TB of RAM — theoretically possible, but expensive and often wasteful.
2. Page replacement policies matter. In OS terms, deciding which documents to retrieve is like choosing which pages to swap in. A good retriever is like a good page replacement algorithm — it predicts what the "processor" (LLM) will need next.
3. Foregoing retrieval entirely is like storing all files in RAM instead of disk. Cost-wise, it makes no sense at scale — even as RAM gets cheaper, disk will always be cheaper still.

Libraries like MemGPT (now Letta) and mem0 implement this memory management pattern explicitly, enabling LLMs to maintain long-term conversational memory across sessions by swapping relevant history in and out of context.

RAG isn't just for knowledge retrieval. Here are two powerful alternative applications — plus important limitations to be aware of:

RAG isn't limited to text documents. Two important extensions:

Multimodal RAG

If your generator is multimodal, you can augment its context with images, videos, and audio — not just text. For example, asking "What's the color of the house in the Pixar movie Up?" — the retriever can fetch an actual image of the house to help the model answer.

For metadata-based retrieval, images are found via their titles, tags, and captions. For content-based retrieval, you need a multimodal embedding model like CLIP (Radford et al., 2021) that produces embeddings for both images and text in the same vector space. The workflow: (1) generate CLIP embeddings for all images and texts, (2) for a query, generate its CLIP embedding, (3) find nearest neighbors across both modalities.

Tabular RAG (Text-to-SQL)

When answers live in database tables rather than documents, the RAG workflow changes significantly. Imagine an e-commerce site with an orders table. The query "How many Fruity Fedora hats were sold in the last 7 days?" requires:

1. Text-to-SQL: The LLM translates the natural language query into SQL based on the provided table schemas
2. SQL execution: The generated query is run against the database
3. Generation: The LLM produces a natural language response from the SQL results

-- Generated SQL from: "How many Fruity Fedoras sold this week?"
SELECT SUM(units) AS total_units_sold
FROM Sales
WHERE product_name = 'Fruity Fedora'
AND timestamp >= DATE_SUB(CURDATE(), INTERVAL 7 DAY);

If there are many tables whose schemas don't all fit in the context window, you may need an intermediate step to predict which tables are relevant for each query. The text-to-SQL step can use the same LLM as the final generator or a specialized model.