Explainers · updated July 2026
What is chunking in RAG? The cut that decides what the AI finds.
Before ChatGPT can answer a question about the PDF you just uploaded, something quietly cuts that document into pieces. RAG calls them chunks, and where the cuts land decides what the model can (and can't) find later. Chunk too coarse and retrieval drowns the answer in noise; chunk too fine and it slices a single idea in half. It is the most consequential setting in a Pinecone-style RAG stack, and the one most people never touch. Here is what chunking is, why its size is a real tradeoff, and where it quietly falls apart on code.
TL;DR · the short version
Chunking, in five lines.
Chunking is splitting a document into smaller pieces (chunks) so each one is small enough to embed as a single vector and be retrieved on its own.
RAG needs it because a whole file is usually too big to embed meaningfully or to fit a retrieval budget. The vector store holds chunks, not files.
Chunk size is a hard tradeoff: small is precise but context-poor, large is context-rich but noisy. There is no universal best size.
Chunks usually overlap (a shared band of roughly 10 to 20%), so an answer that lands on a boundary still survives in at least one chunk.
Chunking degrades on code: functions, imports and call sites get split, and exact strings blur into fuzzy matches. For literal, up-to-the-second lookups, pair RAG with an exact index.
The mechanic
One document. Many little vectors.
Chunking is the step in a RAG pipeline that splits a document into smaller pieces (chunks) so each one is small enough to turn into a single embedding and be retrieved on its own. It runs once, offline, before any question is asked.
Why cut at all? An embedding compresses a passage into one fixed-length vector: the longer the passage, the more distinct meaning gets averaged into blur. Retrieval also works on a budget: it fetches a handful of chunks to paste into the prompt, not whole files. So the pipeline slices every document into consistent, embeddable pieces. And because a naive cut can fall in the middle of the exact sentence that holds the answer, each chunk is given an overlap (a shared band of text at every seam), so the answer survives in at least one chunk.
Every chunk becomes one vector in the store. At query time, retrieval compares your question to those vectors and pulls back the nearest few, so the granularity you choose here decides what the model is even allowed to see. This is one stage of the larger flow in what is RAG, and each chunk is turned into a vector exactly as described in what are embeddings for code.
The hard part
Small or large? Both lose something.
The size of a chunk is the single biggest knob in RAG, and it has no free setting. Shrink it and each hit is sharp but starved of context. Grow it and each hit carries context but blurs the embedding and drags in noise. There is no universal best chunk size (a point Pinecone-style vector-database guides make repeatedly) because the right size depends on your documents and your questions.
| Chunk size | Good at | What it loses | Common use |
|---|---|---|---|
Small ~128 tokens | Sharp, focused hits · low noise in the prompt | Context: one idea gets split across many chunks | Fact lookup, short Q&A |
Medium ~256 to 512 tokens | The usual balance of context and precision | Can still cut a long function or section in half | General docs, the common starting point |
Large ~1,024 to 2,048 tokens | Keeps whole sections together · more context per hit | Blurred embeddings · noisier matches · fewer distinct chunks | Long-form prose, narrative documents |
Most teams start near 256 to 512 tokens with a small overlap and tune from there. But notice what every row quietly assumes: that a clean, meaningful cut exists at all. For prose, it usually does. For code, it usually doesn't.
The failure mode
Code doesn't chunk. It connects.
A paragraph is roughly self-contained. A function is not. Code is a web of references (a symbol defined here, imported there, called in a third place), and chunking cuts straight through those threads. Four things go wrong.
Definitions split from uses. A function's body lands in one chunk; the dozen places that call it land in others. Ask “where is this used?” and retrieval sees only fragments of the answer.
Imports orphaned from code. The import block sits at the top of the file, in its own chunk. The line that depends on it sits 400 lines down, in another. A nearest-neighbour match never sees the connection.
Exact things go fuzzy. An error string, a variable name, a config key: the moment it becomes a vector, it is matched by resemblance, not identity. Similarity search is the worst tool for “find this exact token.”
Every edit needs a re-embed. Rename a symbol and the chunk that mentioned it is stale until someone rebuilds the vectors. Between re-embeds, retrieval quietly answers from the old cut.
None of this means RAG is wrong for code: it means chunking is doing a job it was never shaped for. For the literal, exact, right-now questions agents ask constantly, you want a different lane entirely, laid out in RAG vs index vs grep.
The other half of retrieval
Chunk for meaning. Index for the exact word.
Chunking and exact retrieval answer different questions, so they belong in the same stack, not against each other. Keep chunks and embeddings pointed at meaning, where fuzzy recall is the whole point. For the literal thing (a name, a symbol, a string, an error message), add an always-fresh index that never chunks, never embeds, and never drifts. It searches whole files across every repo on the machine, including the ones git ignores, and it is fresh to your last keystroke.
| The tool | The task | Time | vs Interlinked |
|---|---|---|---|
| Interlinked (live index) | one content query, same corpus | 16 ms | baseline |
| ripgrep (what your agent runs) | same query, same machine | 93.8 s | 58,625× |
Same content query, same files, same machine: 93.8 s on ripgrep → 16 ms on the live index. The full write-up is in content search vs ripgrep. Because the disk was already walked, the query answers before a fresh walk would even begin.
Methodology: Ryzen 9 9950X3D · 64 GB DDR5 · NVMe · Windows 11 · 4,470,000 files. Filename figures are medians: 139 μs across 20 queries, 85 μs single-file; content queries collapse a 93.8 s ripgrep run to 16 ms. At rest the engine sits near 44 MB in Task Manager, and the on-disk index is under 1% of the drive.
CHUNK FOR MEANING · INDEX FOR THE EXACT WORD · A COMPLETE STACK NEEDS BOTH.
Common questions
Chunking, answered.
What is chunking in RAG?
Chunking is the step in a retrieval-augmented generation pipeline that splits a document into smaller pieces (chunks) so each one is small enough to turn into a single embedding and be retrieved on its own. The vector store holds these chunks, and retrieval pulls back the ones nearest to your question.
What is the best chunk size for RAG?
There is no universal best size: it depends on your documents and your questions. Small chunks (around 128 tokens) are precise but lose context; large ones (1,000+ tokens) keep context but blur the embedding and add noise. Most teams start near 256 to 512 tokens with a small overlap and tune from there.
Why do chunks overlap?
Because a clean cut can land in the middle of the exact sentence that answers a question. Giving each chunk a shared band of text with its neighbour (commonly 10 to 20% overlap) means an answer that falls on a seam still appears whole in at least one chunk.
Does chunking work well for code?
Not really. Code is a web of references (a symbol defined here, imported there, called in a third place), and chunking cuts straight through those threads. Definitions get split from their uses, imports from the lines that need them, and exact tokens turn fuzzy. For code, pair meaning-based retrieval with an exact, always-fresh index.
Do I still need chunking if I have an exact index?
Yes. They do different jobs. Keep chunking and embeddings for conceptual, meaning-based questions where fuzzy recall is the point. Add an exact index for literal, right-now lookups (a name, a symbol, a string, an error message) that never chunks, never embeds, and never drifts.
Do this today
Keep chunking. Add the exact lane.
Keep your RAG stack. Chunks, embeddings and the vector database stay exactly where they are, the right tools for meaning-based recall across large, mixed corpora.
Add the exact index. One signed installer auto-configures 19 AI clients: Claude Code, Cursor, Copilot, Codex, Windsurf, Zed, Cline and the rest. No JSON to edit.
Point each at its job. RAG for “what is this about.” The exact index for names, symbols, strings and error messages, whole-machine, across every repo, fresh to the last save.
Never re-embed to stay current. Save a file and it is searchable in about a millisecond. No re-chunk, no re-index, no drift window. At rest the engine sits near 44 MB, less than a browser tab.
Everything local is free forever, on all your devices, no card. Hosting starts at $5.99/mo. Download for Windows.