TL;DR

• Chunking determines retrieval quality more than your choice of embeddings.

• Fixed-size chunking fails silently in production by severing context.

• Structure-aware chunking is the default for technical documentation.

• Evaluate retrieval at the chunk level (Context Recall/Precision), not just the final answer.

• Use MMR (Maximal Marginal Relevance) instead of manual deduplication to handle overlaps.

Most RAG systems don’t fail because of the model. They fail because of how the data is chunked.

The Failure That Started This

In one system I worked on where Precision looked fine in evals, but engineers started complaining that answers were wrong.

The root cause: a Python function definition—signature, docstring, and implementation was split across two chunks at a 512-token boundary. The retriever found the signature chunk, but the logic needed to answer the query was in the second chunk, which was too far away in embedding space to be retrieved. This led to the model hallucinating the missing context with complete confidence.

Chunking failures are insidious because they don't throw errors the system keeps running while answers quietly degrade.

Why Chunking Is the Highest-Leverage Decision in a RAG Pipeline

Most teams optimize in the wrong order, swapping embedding models before fixing chunking. Chunking determines the units of meaning your retrieval system works with. Every downstream component (retrievers, rerankers, and the LLM) is bounded by the quality of your chunks.

There are two critical failure modes:

• Precision failure: Chunks are too large and contain multiple independent ideas, leading to noise.

• Context integrity failure: Chunks are too small, causing a single coherent thought to be severed.

Strategy 1: Fixed-Size Chunking

What it is

Splitting text every \(N\) tokens with \(M\) tokens of overlap.

from langchain.text_splitter import RecursiveCharacterTextSplitter
import tiktoken

def token_length(text: str, model: str = "gpt-4") -> int:
    enc = tiktoken.encoding_for_model(model)
    return len(enc.encode(text))

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=51,
    length_function=token_length, 
    separators=["\n\n", "\n", ". ", " ", ""]
)

💡 Fixed-size chunking optimizes for simplicity, not meaning.

When it works

• Narrative prose where meaning flows continuously.

• Building an initial baseline for reproducibility.

When it quietly fails

• Split code signatures: Cutting straight through function names.

• Table destruction: Splitting Markdown/HTML tables makes them semantically meaningless.

Strategy 2: Semantic Chunking

What it is

Embedding every sentence and splitting where the embedding distance between adjacent sentences "spikes."

💡 You are trading compute at indexing time for retrieval quality at query time.

When it works

• Technical documentation where topic shifts are semantically clear.

• Multi-topic documents where fixed-size chunking would straddle unrelated concepts.

When it quietly fails

• Indexing Cost: High latency due to embedding every sentence before indexing.

• Size Inconsistency: Chunks can range from one sentence to several paragraphs.

Strategy 3: Document-Structure-Aware Chunking

What it is

Parsing structural elements like headings, code blocks, and tables to define boundaries.

from langchain.text_splitter import MarkdownHeaderTextSplitter

headers_to_split_on = [("#", "h1"), ("##", "h2"), ("###", "h3")]
splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
chunks = splitter.split_text(markdown_doc)

💡 This is the default choice for production systems handling technical content.

When it works

• Corpora with predictable structure (READMEs, API docs).

• Metadata Power-Up: Hierarchy metadata (e.g., {"h1": "Auth"}) allows for metadata filtering, reducing search space by 40-60%.

Strategy 4: Late Chunking

What it is

Instead of splitting first and embedding later, you embed the entire document first and then chunk the embeddings. This ensures each individual chunk retains the full document's context.

When it works

• Long technical documents with dense cross-references and pronouns.

When it quietly fails

• Infrastructure Lock-in: Requires specific models (like Hugging Face TEI) that expose hidden states.

• Context Limits: Limited by the embedding model's maximum input window (e.g., 8k tokens).

Strategy 5: Proposition-Based (Atomic Fact) Chunking

What it is

Using an LLM to decompose text into atomic, self-contained factual statements.

When it works

• High-precision QA systems where a wrong answer is unacceptable.

When it quietly fails

• LLM Cost: Decomposing large corpora can be significantly more expensive than standard indexing.

• Loss of Sequence: Procedural or step-by-step content loses ordering and context.

The System Design Decision Matrix

The Failure You Haven't Thought About Yet: Eval Debt

If you’re not measuring context recall and precision, you are debugging blind. End-to-end evals mask chunking failures. You need chunk-level metrics via frameworks like Ragas:

• Low Context Recall: You are missing relevant chunks (context integrity failure).

• Low Context Precision: Relevant info is buried in noise (chunks are too large).

The Overlap Trap

Overlap causes redundant retrieval. Use Maximal Marginal Relevance (MMR) in your vector database to maximize both relevance and diversity in the results.

Conclusion

Chunking is infrastructure, not configuration.

The choice you make at indexing time defines the ceiling of your entire RAG system. Get it wrong, and no amount of model tuning will save you. Get it right, and everything downstream becomes easier.

Get it wrong, and your retriever will lie to you quietly.

About the Author

Parikshit is a Senior AI/ML Engineer specializing in Generative AI, LLM systems, and MLOps. He focuses on building scalable API architectures and optimizing high-throughput inference pipelines. GitHub · LinkedIn

In this post, we explore the real challenges of building scalable and reliable RAG systems in production environments.

TL;DR

Most RAG tutorials stop at the happy path.Production systems fail in the unhappy path.

Here’s what actually breaks in production:

• Fixed-size chunking reduces retrieval quality

• Measure retrieval separately (Precision@K, Recall@K)

• Add hallucination validation (NLI)

• Track retrieval and generation latency independently

• Production RAG is a systems problem, not just an LLM problem

Every RAG tutorial shows you the same thing:

# The tutorial version
chunks = split_text(docs, chunk_size=512)
embeddings = embed(chunks)
results = vector_db.search(query, top_k=5)
answer = llm.generate(query, context=results)

This works in a notebook. It fails in production.

Here’s what the tutorials don’t tell you.

Problem 1: Why Fixed-Size Chunking Fails in RAG Systems

512-token chunks are arbitrary. They:

• Cut sentences mid-thought

• Separate headings from content

• Break semantic meaning

What actually works: semantic chunking

from sentence_transformers import SentenceTransformer
import numpy as np

def semantic_chunk(text, model, threshold=0.3):
    sentences = split_into_sentences(text)
    embeddings = model.encode(sentences)

    chunks, current_chunk = [], [sentences[0]]

    for i in range(1, len(sentences)):
        similarity = cosine_similarity(
            embeddings[i - 1].reshape(1, -1),
            embeddings[i].reshape(1, -1)
        )[0][0]

        if similarity < threshold:  # semantic boundary
            chunks.append(" ".join(current_chunk))
            current_chunk = []
        current_chunk.append(sentences[i])

    if current_chunk:
        chunks.append(" ".join(current_chunk))
    return chunks

Impact: +23% retrieval accuracy improvement on held-out eval.

Problem 2: Missing Retrieval Evaluation in RAG Systems

Most teams evaluate RAG systems by asking:

“Does the final answer look correct?”

This is misleading.

A correct-looking answer doesn’t guarantee:

• The retrieved context was relevant
• The model didn’t compensate with its own knowledge
• The system will generalize to new queries

In practice, retrieval and generation fail in different ways.

Without measuring retrieval quality separately, you have no visibility into where the system is breaking.

Measure retrieval directly:

def evaluate_retrieval(eval_set, retriever, k=5):
    """
    eval_set: list of {query, relevant_chunk_ids}
    """
    precision_scores, recall_scores = [], []

    for item in eval_set:
        retrieved = retriever.search(item['query'], top_k=k)
        retrieved_ids = {r.id for r in retrieved}
        relevant_ids = set(item['relevant_chunk_ids'])

        precision = len(retrieved_ids & relevant_ids) / len(retrieved_ids)
        recall = len(retrieved_ids & relevant_ids) / len(relevant_ids)

        precision_scores.append(precision)
        recall_scores.append(recall)

    return {
        'precision_at_k': np.mean(precision_scores),
        'recall_at_k': np.mean(recall_scores)
    }

Track this metric independently from generation quality. They fail in different ways.

Problem 3: No Hallucination Guardrails in LLM Systems

RAG doesn’t eliminate hallucination- it shifts where it happens.

Instead of the model inventing facts, it can:

• Misinterpret retrieved context
• Over-generalize from partial information
• Confidently answer questions with weak grounding

This makes hallucination harder to detect because the answer looks correct.

Add NLI-based validation post-generation:

from transformers import pipeline

nli = pipeline(
    "text-classification",
    model="cross-encoder/nli-deberta-v3-base"
)

def validate_answer(answer, source_chunks, threshold=0.7):
    premise = " ".join(source_chunks)
    result = nli(f"{premise} [SEP] {answer}")

    entailment_score = next(
        r['score'] for r in result
        if r['label'] == 'ENTAILMENT'
    )

    if entailment_score < threshold:
        return {
            "answer": answer,
            "confident": False,
            "message": "Answer may not be fully supported by sources",
            "entailment_score": entailment_score
        }

    return {"answer": answer, "confident": True}

Problem 4: Most teams track total latency:

retrieval + generation + network + processing

But this hides the real bottleneck.

In production systems:

• Retrieval latency depends on indexing and search
• Generation latency depends on model and prompt size

Without separating them, you can’t debug performance issues effectively.

import time
from dataclasses import dataclass

@dataclass
class RAGLatencyBudget:
    retrieval_ms: int = 40
    generation_ms: int = 60

def rag_with_budget(query, retriever, generator, budget=RAGLatencyBudget()):
    t0 = time.time()
    chunks = retriever.search(query)
    retrieval_ms = (time.time() - t0) * 1000

    t1 = time.time()
    answer = generator.generate(query, chunks)
    generation_ms = (time.time() - t1) * 1000

    # Log separately — they fail differently
    metrics.record('retrieval_latency_ms', retrieval_ms)
    metrics.record('generation_latency_ms', generation_ms)

    if retrieval_ms > budget.retrieval_ms:
        alerts.warn(f"Retrieval exceeded budget: {retrieval_ms:.0f}ms")

    return answer

The Checklist Before You Ship

• [ ] Semantic chunking (not fixed-size)

• [ ] Retrieval precision measured separately (Precision@K, Recall@K)

• [ ] NLI-based hallucination validation

• [ ] Retrieval + generation latency tracked independently

• [ ] Adversarial eval set (trap questions)

• [ ] “I don’t know” path implemented

Final Thought

The model is the easy part.

Production RAG systems don’t fail because of LLMs. They fail because of retrieval, evaluation, and system design.

If you're building RAG systems in production, what challenges have you faced?

Let’s discuss 👇