Agentic Data Engineering Resources

Resource

Chunking Strategies for RAG and LLMs: How to Improve Retrieval Quality

Compare chunking strategies: fixed-size, semantic, late chunking, contextual retrieval. Learn how to choose the right one for RAG and AI agents.

Airbyte Engineering Team

June 24, 2026

Summarize with AI:

Retrieval quality in Retrieval-Augmented Generation (RAG) systems depends on the text unit you embed and retrieve. Chunking splits documents into segments before embedding, so it directly shapes that unit.

For AI agents that manage context budgets across multi-step workflows, poor chunk boundaries compound errors. A chunk that is too large wastes tokens. One that is too small loses meaning. A split at the wrong boundary can trigger an incorrect tool call or force hallucination.

TL;DR

Chunking affects retrieval quality by controlling the text units that are embedded and retrieved, but it is one factor among embedding model choice, retrieval method, and reranking.
No chunking strategy performs best in every setting; results depend on document structure, domain, embedding model, and retrieval configuration.
Data quality before chunking, including parsing, structure preservation, and freshness, often matters more than switching between chunking strategies.
For most teams, recursive character splitting with moderate chunk sizes and overlap is a strong starting point, followed by benchmarking on their own corpus.

What Is Chunking and Why Does It Matter for Retrieval Quality?

Chunking is the process of splitting a document into smaller segments, with each segment embedded into a vector representation and stored for retrieval. The size, shape, and boundaries of those segments determine what the retriever can actually find at query time.

A NAACL 2025 study suggests that embedding quality often overshadows the choice of chunking strategy, but chunking still sets the ceiling on what any retriever can return.

Chunking matters for retrieval quality because:

It controls semantic precision. A large, multi-topic chunk produces an averaged embedding that represents no single topic well, so cosine similarity against a specific query drops. Smaller, topic-focused chunks yield vectors that match precise queries more accurately.
It interacts with embedding model token limits. Models like BGE v1.5 cap at 512 tokens; BGE-M3 supports 8,192 tokens; and Cohere's embed-v4.0 reaches 128,000 tokens. Content beyond the limit is truncated or rejected, and effective retrieval often degrades well before the advertised maximum is reached.
It determines whether retrieved units are self-contained. Chunks split at the wrong boundary lose the context needed for downstream generation, forcing the model to guess or hallucinate.
It governs context budget efficiency. Oversized chunks waste tokens that agents and RAG pipelines need for tool responses, reasoning, and additional retrieved evidence.

What Are the Main Chunking Strategies?

Chunking strategies fall into two broad families. Rule-based approaches rely on deterministic logic such as character counts, separators, or document structure. Model-dependent approaches use embeddings or LLMs to make boundary decisions based on the content itself.

Rule-Based Strategies

Fixed-size chunking splits text at predetermined intervals with optional overlap. It has no semantic awareness but runs at minimal cost and serves as the baseline against which other methods are measured.
Recursive character splitting applies a prioritized separator hierarchy (paragraphs, line breaks, spaces, characters) to respect natural boundaries when possible. LangChain docs describe it as the default text splitter for generic use cases.
Sentence-based and structure-based chunking split at sentence boundaries using NLP tokenizers or at document elements such as headers, sections, and code blocks. Sentence splitting works well for prose; structural splitting works best when the source has consistent markup.

Model-Dependent Strategies

Semantic chunking embeds individual sentences and groups consecutive ones whose embeddings exceed a cosine similarity threshold, with that threshold acting as a sensitive hyperparameter.
Late chunking and contextual retrieval add document-level context to each chunk. Late chunking encodes the full document first and then pools token embeddings per chunk, while contextual retrieval, introduced by Anthropic, prepends an LLM-generated 50–100-token summary and reports a 49% reduction in failed retrievals (67% with reranking).
Agentic chunking uses an LLM to decide chunk boundaries and content, producing semantically self-contained segments at the cost of many model calls per document.

These families span a spectrum from simple string operations to full LLM inference per chunk, and most production systems mix elements from both. The table below summarizes the eight strategies most commonly used in practice so you can compare cost, precision, and best-fit document types side by side.

Eight Main Chunking Strategies Overview

The matrix below maps each strategy to the document types it handles best, the computational cost it incurs, and the trade-offs you should expect when deploying it. Use it as a quick reference when scoping a new RAG pipeline or revisiting an existing one.

Strategy	How It Works	Best Document Types	Computational Cost	Retrieval Precision	Key Limitation
Fixed-size (token or character)	Splits text at fixed intervals with optional overlap	Homogeneous text, general-purpose corpora	Low	Moderate; often competitive with semantic methods	Splits mid-sentence, breaking semantic units
Recursive character splitting	Tries progressively smaller separators	Mixed-format text, general-purpose	Low	Moderate-high; de facto standard across major frameworks	Cannot detect topic shifts within a paragraph
Sentence-based	Splits at sentence boundaries using NLP tokenizers	Q&A systems, conversational data, call transcripts	Low-moderate	Moderate; preserves atomic semantic units	Individual sentences may lack sufficient context
Document structure-based	Splits at headers, sections, or page boundaries	PDFs, HTML, Markdown, code files	Low-moderate	High for well-structured documents	Requires consistent structural markup
Semantic (embedding similarity)	Groups text by embedding similarity at topic boundaries	Multi-topic documents, research papers	Moderate-high	Variable: depends on document type and domain	Over-fragmentation risk; computational overhead
Late chunking	Embeds full document first, then splits embedding sequence	Long documents requiring cross-reference context	Moderate (requires long-context embedding models)	Small but consistent improvement over naive methods	Requires models supporting mean pooling and long context
Contextual retrieval	Prepends LLM-generated summary to each chunk before embedding	Complex enterprise documents, legal, compliance	Moderate (with prompt caching)	Reduces retrieval failures significantly; amplified by reranking	Summaries must regenerate when sources change
Agentic (LLM-driven)	An LLM agent decides how to split and annotate each document	Regulatory filings, multi-section contracts, corporate policies	Very high (multiple LLM calls per proposition)	Highest potential for complex documents	Currently impractical at scale

How Does Chunking Affect AI Agent Performance?

Agents that reason across multiple sources and manage context window limits in multi-step workflows consume chunks differently than single-turn Q&A systems do. Three mechanisms connect chunking decisions to agent reliability.

Context Window Budget Management

In a single-turn RAG system, the context window is consumed once per call and resets. In an agent loop, the context window becomes a cumulative, multi-step resource. Every retrieved chunk, tool response, conversation history entry, and intermediate reasoning step competes for the same budget across all future iterations of the loop.

Oversized chunks consume budget that agents need for tool responses and intermediate reasoning. If each retrieval step returns five chunks at 1,000 tokens each, three retrieval steps consume 15,000 tokens of context. The same three steps with 400-token chunks consume 6,000 tokens, which frees 9,000 tokens for tool responses and reasoning. Over a multi-step workflow, this difference compounds.

Chunk Self-Containment for Tool-Call Accuracy

A chunk that cannot be understood without its surrounding context creates a specific agent failure: the agent populates tool arguments from incomplete information. If a chunk says "apply the rate from the standard schedule" and the standard schedule is in an unretrieved chunk, the agent may call a pricing tool with a fabricated parameter value.

When the context contains incomplete information, the model may select the wrong tool even when the correct option exists in its toolkit. Chunks that are too large create a different problem. Mixed-topic content produces embeddings that average across multiple subjects, and retrieval returns chunks that are weakly related to the query on multiple dimensions at once. The agent receives noisy context and may synthesize across irrelevant details.

Metadata enrichment, such as source system, entity type, and timestamp, matters more for agents than for static Q&A because agents use metadata to plan multi-step retrieval sequences. An agent deciding whether to query a CRM, a support system, or a billing platform benefits from metadata that identifies the origin system for each chunk. In agentic RAG systems, each chunk should function as a self-contained semantic unit that the agent can selectively access through dedicated retrieval interfaces.

Why Does Data Quality Matter Before Chunking?

Pre-chunking parsing quality sets the ceiling for chunking. If the data entering the chunker is incorrect, no strategy can fix it downstream. The most common upstream failures include:

Table structure loss: PDF parsers that extract text linearly flatten row-column relationships, and even top-performing LLM-based parsers misalign spanning cells or misread headers. No chunker reconstructs the original table from linearized text.
Multi-column layout interleaving: Parsers stitch together content from parallel columns, producing incoherent text that semantic chunkers misread as coherent, while structural chunkers cannot segment it.
OCR and encoding errors: Corrupted tokens like "Artlcle 4.2" never match a query for "Article 4.2" because the embedding vectors diverge at the character level.
Unnormalized SaaS payloads: Raw API responses with nested JSON and inconsistent field names create noisy chunks; CRM records, tickets, and analytics exports need extraction and normalization first.
Stale or conflicting content: Even when retrieval surfaces the correct document, conflicting figures can cause errors leading to incorrect outputs.

Convert PDFs to a structured format, such as Markdown, before chunking and treat parsing quality and data freshness as prerequisites. In practice, chunking only helps when the underlying context is clean, current, and structured.

What Are the Best Practices for Chunking?

Once data quality is in place, the following practices help you ship a retrieval pipeline that performs predictably and improves over time.

1. Start With a Strong Default Configuration

Recursive character splitting at 400 to 512 tokens with 10 to 20% overlap is the strongest general-purpose default. NVIDIA found that medium-sized chunks in the 512 to 1,024 token range showed optimal retrieval performance, with performance degrading at 2,048 tokens. Use smaller chunks for code and technical content and somewhat larger ones for narrative material.

For deeper tactical guidance, see chunking best practices.

2. Measure With the Right Retrieval Metrics

Establish a baseline using a fixed set of evaluation queries, and change one variable at a time. Four metrics diagnose chunking problems directly:

Low contextual relevance suggests that chunks are too large or that top-K is too high.
Low contextual recall suggests chunks are too small or boundary splits are separating relevant content.
Low faithfulness suggests chunks lack sufficient context for accurate generation.
Low hit rate (Recall@K) means relevant content is not being retrieved at all.

Sweep multiple chunk sizes (128, 256, 512, 1,024, 2,048) at 20% overlap, tracking nDCG@K and MRR. Cover both offline experiments and ongoing production monitoring.

3. Add Reranking After Chunking Is Stable

Reranking often delivers higher return on engineering effort than switching chunking strategies. An arXiv benchmark found that reranking yielded the largest single-component improvement: +17.2 percentage points MRR@3 and +12.1 percentage points Recall@5. Reranking can only reorder what was retrieved, so fix chunking first and use reranking to amplify it.

4. Enrich Chunks With Metadata for Agent Workflows

Attach the source system, entity type, document ID, section path, and timestamp to every chunk. Metadata enables filtered retrieval, hybrid search, and per-source ranking, and it lets agents plan multi-step retrieval sequences across CRMs, ticketing systems, and knowledge bases. Store metadata in the vector index payload so it travels with the embedding at query time.

5. Solve the Data Integration Layer Before Chunking

Most retrieval problems show up as bad chunks but originate upstream, in raw API payloads, inconsistent field names, and stale extracts from SaaS systems. Pre-materializing normalized records before chunking removes the need to stitch together fragmented API responses at runtime.

Airbyte Agents handle this layer via the Context Store, a unified, searchable repository of curated, normalized records that downstream chunking and retrieval pipelines can rely on. The Context Store pre-materializes data into curated, normalized records instead of leaving agents to parse raw API payloads at runtime, while direct API access is reserved for live state and writes.

What Is the Most Effective Way to Improve Retrieval Quality Through Chunking?

Retrieval quality compounds from clean upstream data, a chunking approach matched to document structure and consumption pattern, and iteration against your own retrieval metrics rather than published benchmarks. For agents specifically, chunk sizing and self-containment matter more than for static Q&A because every token competes in multi-step reasoning loops, and weak boundaries surface as bad tool calls.

Airbyte Agents gives agent builders that clean foundation, with connectors, metadata enrichment, and pre-materialized retrieval that drop structured records straight into RAG pipelines and agent workflows.

Builders can interact with the platform through the Web app, Agent MCP, Airbyte's Agent SDK, and API, with additional surfaces including the MCP Gateway for routing and governance and the Agent CLI for command-line workflows.

Talk to sales to see how Airbyte Agents fits your architecture, or try Airbyte Agents and start building against the Context Store today.

Explore the For Developers hub for reference implementations.

Frequently Asked Questions

How Do I Handle Chunking for Code and Technical Documentation?

Code benefits from structure-aware splitting that respects function, class, and module boundaries rather than character intervals. Tree-sitter parsers or AST-based splitters preserve syntactic units, so retrieved chunks remain executable or referenceable. For mixed prose and code, use a hierarchical approach that keeps surrounding explanation attached to the relevant block. Smaller chunk sizes (around 256 to 512 tokens) usually work better than long ones because functions and code blocks tend to be self-contained.

How Do I Evaluate Chunking Quality Without Ground-Truth Labels?

Start with synthetic query generation: prompt an LLM to write realistic questions from each document, then measure whether retrieval surfaces the originating chunks. Combine this with proxy metrics such as answer faithfulness, citation overlap, and human spot checks on production queries. LLM-as-judge evaluations can score relevance at scale, though calibration against a small human-labeled sample is recommended. Track these signals over time so regressions surface quickly.

How Often Should I Re-Chunk My Document Collection?

Re-chunk whenever source documents change materially, when you swap embedding models, or when you adjust chunk size and overlap parameters. For incremental updates, chunk only the affected documents and write new vectors alongside existing ones, with versioning so older embeddings can be deprecated cleanly. Full re-indexing is typically scheduled quarterly or on major model upgrades. Real-time pipelines should re-chunk on document update events to keep retrieval current.

Does Chunk Overlap Always Improve Retrieval?

Overlap helps when boundary information would otherwise be lost between adjacent chunks, which is common in narrative or argumentative text. It hurts when the corpus is highly structured because duplicate content inflates index size and can distort rank-based metrics. A 10 to 20% overlap is a reasonable default for prose, while structured documents like API references or legal sections often perform better with little or no overlap. Test both settings on a representative query set before committing.

Try Airbyte Agents

Airbyte connects your agents to all of your data and assembles context before they run. Build agents that actually know your business.

Try it free Talk to sales

Chunking Strategies for RAG and LLMs: How to Improve Retrieval Quality

Related posts

Try Airbyte Agents