RAG Chunking and Embedding Model Choice

Quick answer

RAG chunking and embedding model choice should be made from your corpus shape and gold questions, not from a universal chunk-size recipe. Start with structure-aware chunks for headings, slides, tables, and code; use fixed token windows only when the corpus is mostly plain prose. Choose an embedder for language coverage, domain vocabulary, vector dimension, license, and latency, then measure recall@k on labeled question-passage pairs before tuning prompts or swapping LLMs. Add a reranker only when false positives remain after recall is healthy.

Key takeaways

• Chunk boundaries dominate perceived embedding quality more often than the last 2% on a public leaderboard.
• Multilingual corpora need multilingual embedders—or language-specific indexes—not English-only defaults.
• Parent-child chunking pairs summary nodes with detail nodes for better two-stage retrieval.
• Rerankers improve precision at latency and cost; use them when false positives hurt UX.
• Log chunk IDs in citations so retrieval failures are debuggable (retrieval eval guide).

Prerequisites

• A representative document sample (PDF, HTML, tickets, code—not only polished marketing pages).
• 200+ real or synthetic user questions with annotated gold passages (expand over time).
• Vector database or search service chosen with dimension and distance metric documented.
• Baseline generator model for end-to-end faithfulness checks—not for retrieval tuning alone.
• Access control requirements if multi-tenant (RAG pillar).

Step 1: Audit your corpus shapes

Before picking chunk sizes, inventory formats and failure modes:

Desk method note: we compare chunk strategies by holding the embedder and search params fixed, then swapping chunk pipelines—otherwise teams attribute chunk wins to the wrong embedder upgrade.

Step 2: Choose a chunking strategy

Fixed token windows with overlap

Split text into windows of N tokens (typical sweeps: 256, 512, 768, 1024) with overlap (often 10–20% of N) to preserve sentences spanning boundaries. Works for homogeneous prose when structure parsing is expensive. Weak on tables, code, and legal numbering unless preprocessed.

Markdown and HTML header splits

Split on heading hierarchy; merge small sections until minimum token count; split oversized sections recursively. Preserves semantic boundaries in wikis, docs sites, and Notion exports. Pair with canonical URL metadata for citations.

Parent-child (hierarchical) chunks

Index small child chunks for precision retrieval; attach each child to a larger parent summary for generator context. Microsoft-style “small-to-big” retrieval reduces noise when users need surrounding paragraph or section context. Store parent_id on every child for prompt assembly.

Specialized parsers

• PDF: Layout-aware extraction (not raw text dump) for columns and footnotes.
• Code: Tree-sitter or language-aware splitters; index docstrings separately from implementations if queries are conceptual.
• Tables: Serialize rows as JSON lines or markdown tables; never split mid-row without headers repeated.

Chunk size sweep protocol

1. Fix embedder and k=10 retrieval.
2. Run recall@5 and recall@10 on gold set for each chunk config.
3. Inspect 20 failures manually—label “chunk boundary” vs “embedder” vs “missing doc.”
4. Pick the smallest average chunk size that hits recall SLO to save prompt tokens downstream.

There is no universal best chunk size. Public blogs citing “512 always wins” rarely disclose corpus type. Sweep on your eval set.

Step 3: Select an embedding model

Open vs API embedders

Open weights (self-hosted sentence-transformers class models, bge/e5/nomic families on Hugging Face) offer control, air-gap deployment, and predictable unit economics at scale—see inference economy. You own reindex timing and hardware.

Commercial embedding APIs bundle quality and ops; watch vendor lock-in, dimension changes, and rate limits. Some providers couple embed + generate—convenient but harder to swap one layer independently.

Selection criteria (in order)

1. Language coverage matching user queries and documents.
2. Domain fit (legal, medical, code) via small offline eval—not leaderboard rank alone.
3. Vector dimension compatible with your index and memory budget.
4. Latency and throughput at your QPS for query embedding (and document embedding at ingest).
5. License for commercial use and redistribution if self-hosting (M6).

Use the public MTEB embedding leaderboard to shortlist candidates, then run recall@k on your gold set. Leaderboard tasks differ from your ticket phrasing.

Multilingual and cross-lingual setups

If users query in Spanish but docs are English (or mixed), verify cross-lingual retrieval on held-out pairs. Options include multilingual embedders, translated indexes (expensive), or language-specific indexes with query routing. Do not assume English-only e5/bge variants cover code-switching support chats.

When to re-embed

• Embedding model version change (even minor).
• Chunking pipeline change affecting text boundaries.
• Material document schema change (new mandatory metadata fields in chunk text).

Run blue/green indexes; compare recall before cutover. Document index version in observability logs.

Step 4: Configure search and metadata

Hybrid search

Dense vectors excel at paraphrase; sparse BM25 excels at SKU numbers, error codes, and rare tokens. Hybrid retrieval (weighted or reciprocal rank fusion) is common in enterprise search. Tune weights on the same gold set used for chunk sweeps.

Metadata filters

Attach product, region, doc version, effective date, clearance level. Apply filters before vector search when possible to reduce false positives and ACL risk. Regulated verticals need effective-date filters for legal and finance corpora (vertical RAG).

Distance metric and normalization

Match metric to embedder training (cosine vs dot product). Normalize vectors if your stack expects unit length. Mismatch here silently hurts recall.

Step 5: Add a reranker (optional but common)

First-stage retrieval optimizes recall@k with k often 20–100. Cross-encoder or lightweight rerankers rescore top candidates for precision before LLM context assembly. Tradeoffs:

Add rerankers when users complain about “right topic, wrong paragraph” despite decent recall@20. Skip rerankers for ultra-low-latency FAQ bots if recall@5 is already high.

Step 6: Wire citations and logging

Each chunk needs stable IDs, source URI, version hash, and optional byte offsets. Return these in API responses for UI footnotes and for eval alignment. When faithfulness fails, reviewers open exact chunks—not regenerated summaries.

Agent systems should pass chunk metadata into tool traces (agent map). Session-only memory without chunk IDs is not auditable.

Step 7: End-to-end verification loop

1. Retrieval metrics: recall@5, recall@10, MRR on gold set (full eval guide).
2. Generation metrics: faithfulness to retrieved chunks (human rubric or calibrated LLM judge).
3. Latency: p95 embed + search + rerank + prompt assembly.
4. Regression gates: block deploy if recall drops > agreed threshold on holdout.

Do not tune solely on LLM-as-judge without human spot checks—misaligned judges optimize for fluent wrong answers. Align with leaderboard discipline from how to read AI leaderboards.

Failure modes and fixes

Graph and entity-heavy corpora

When questions require joins across entities (“which vendor supplies part X used in plant Y”), flat chunks may be insufficient. Layer graph retrieval or community summaries per GraphRAG for enterprises. Chunking still matters for document nodes attached to graph entities.

Worked example: documentation RAG (fictional but realistic)

Imagine a B2B SaaS company with 12,000 HTML help pages, 3,000 PDFs, and 400 API reference pages. A practical rollout:

1. Week 1–2: Sample 300 real support questions; annotate gold chunk IDs manually.
2. Week 3: Implement header-based chunking for HTML; layout-aware PDF parsing; AST splits for API code samples.
3. Week 4: Shortlist three embedders from MTEB multilingual slice; run recall@10 on gold set.
4. Week 5: Add BM25 hybrid after recall plateaus on keyword-heavy error codes.
5. Week 6: Add reranker if precision@3 still fails SME review on 50 cases.
6. Week 7+: Wire citation IDs; gate releases on holdout recall and faithfulness sample.

This sequencing avoids the common trap of swapping LLMs while recall@10 remains near random on API paths.

Embedding model families (desk-level map, not a ranking)

Public leaderboards rotate; treat family traits as hypotheses to test—not purchase orders.

Ingest pipeline checklist

• Deduplicate near-identical pages (version churn) before embed to save cost.
• Strip boilerplate nav/footer text that pollutes retrieval.
• Compute content hash per chunk; skip re-embed if unchanged.
• Store source mimetype and parser version for debugging regressions.
• Queue failed parses for human fix—silent skips become recall holes.
• Partition indexes by product and environment (prod vs draft docs).

Query-side preprocessing

User queries are messy: acronyms, SKU formats, misspellings, pasted error stacks. Preprocessing options:

• HyDE-style expansion: Generate hypothetical answer for embed search—can help recall; validate for hallucinated expansion terms.
• Spell-normalization for product names with dictionaries—not LLM guesses.
• Query routing: Send code-like queries to code index; policy questions to policy index.
• Metadata from session: Product SKU, locale, plan tier—apply as filters, not prompt stuffing.

Storage and index operations

Vector DB choice affects ops more than textbook benchmarks: replication, backup, filtering performance, hybrid search support, and multi-tenant isolation models. Dimension changes usually require reindex—not in-place patch. Plan blue/green indexes and traffic cutover windows. Document SLAs for ingest lag after doc publish events—stale index is a freshness bug even with perfect embedder.

Generator context assembly

After rerank, assemble context with deterministic ordering (by score, then doc hierarchy). Include chunk title, section, URL, and effective date in the prompt prefix—models cite better when metadata is explicit. Cap total tokens below generator comfort zone; long-context marketing does not remove quality decay on noisy piles (long context in practice). Leave headroom for user question and answer.

Security during ingest and retrieval

Malicious documents can contain prompt injection strings designed to surface in retrieval. Sanitize where feasible; never execute retrieved HTML; separate instructions from content with template discipline (safety for builders). Log retrieval for incident response when users report odd answers.

Cost notes

Ingest cost scales with corpus token count × embedder throughput. Query cost scales with QPS × (query embed + search + rerank). Smaller chunks increase chunk count and storage but can reduce generator prompt size if retrieval precision improves. Build a token budget tied to top user journeys (inference economy).

Self-hosted embedders on CPU can be sufficient for batch ingest while GPU serves query embed at peak. Coupled API pricing may hide embed cost inside generator bills—split line items when comparing vendors.

Who this is for

• ML and data engineers building first production RAG indexes.
• Search teams modernizing lexical search with vectors.
• Platform teams standardizing chunk + embed pipelines across products.

Who should skip

• Teams without any labeled eval questions—create gold set skeleton first.
• Readers wanting a single embedder brand recommendation without corpus context.
• Organizations skipping ACL design—fix governance before optimizing chunk overlap.

Common mistakes

• Using default PDF text extraction on scanned contracts without OCR QA.
• Indexing boilerplate headers/footers that dominate retrieval.
• Changing embedder without reindexing staging and production in sync.
• Optimizing chunk size on synthetic questions only—real users paraphrase differently.
• Ignoring safety on retrieved content injection paths.

FAQ

What is the best chunk size for RAG?

There is no universal size. Sweep 256–1024 tokens (or structure-aware equivalents) on your labeled eval set and pick the smallest chunks that meet recall SLOs.

Should query and document embedders differ?

Some models train asymmetric query/passage encoders; follow model card instructions. Mixing arbitrary models for query vs doc usually fails.

How many dimensions do we need?

Higher dimension is not automatically better. Match model defaults unless ablation on your data shows gains worth memory cost.

Do we need GraphRAG instead of better chunks?

Fix chunking and hybrid search first. Add graphs when a substantial share of questions needs multi-hop entity reasoning.

When should we fine-tune embedders?

When public embedders plateau on domain recall and you have thousands of in-domain query–passage pairs—otherwise prefer rerankers and chunk fixes (retrain vs retrieve).

How do we handle PDFs with scanned pages?

OCR with quality thresholds; quarantine low-confidence pages for human review. OCR noise destroys embedding geometry—fix upstream before tuning k.

Operational runbook: embedder upgrade

1. Announce freeze on chunker changes during embedder A/B.
2. Build parallel index v2 with new embedder on snapshot corpus.
3. Run recall@5/10 and MRR on dev and holdout sets.
4. Compare p95 query latency and ingest duration vs v1.
5. Shadow traffic: log v2 results without user impact for 48–72 hours.
6. Cut over with rollback pointer to v1 index snapshot.
7. Archive v1 after agreed retention window for incident replay.

Skipping shadow traffic is how teams discover production-only ACL filter interactions that dev sets omitted.

Chunk size sweep worksheet (copy for your runbook)

For each candidate chunk configuration, record: parser type, min/max tokens, overlap, recall@5, recall@10, median chunk count per doc, ingest hours per million tokens, and p95 query latency. Plot recall@10 vs median prompt tokens when rerank top-3 is injected into generator. The winning config minimizes prompt tokens subject to recall SLO—not the config with highest leaderboard anecdote.

Include at least three failure buckets in manual review: boundary split, wrong doc family (metadata filter fix), and embedder semantic miss (try hybrid or reranker). If two buckets dominate, fix ingestion before buying a larger LLM.

Multilingual corpus split strategies

Option A: one multilingual embedder and unified index with language metadata filters. Option B: language-specific embedders and routed indexes—higher ops cost, sometimes better recall. Option C: translate queries to document language at search time—watch translation errors on legal terms. Pick using stratified eval, not engineer fluency alone.

Reranker selection notes

Cross-encoder rerankers score query-passage pairs jointly—accurate but slower. Lightweight rerankers or late-interaction models trade quality for speed. Match reranker training language to corpus. Cap candidate count (e.g., rerank top 30 only) to protect p95 latency under load tests, not idle laptop benchmarks.

Debugging retrieval misses (field guide)

When a gold question misses at k=10, triage in order:

1. Document absent? Ingest gap or ACL over-filter—fix pipeline, not embedder.
2. Document present, wrong chunk? Boundary issue—adjust parser or parent-child linking.
3. Chunk present, low score? Try hybrid BM25, query expansion, or embedder swap on dev.
4. Score high but rerank dropped? Reranker domain mismatch or overly aggressive top-n cut.
5. Retrieval good in logs, answer wrong? Stop chunk work; move to faithfulness eval (eval guide).

Keep a shared spreadsheet of miss archetypes so the team does not re-discover the same PDF footer bug quarterly.

Batch vs streaming ingest

Batch nightly jobs suit stable documentation; streaming or near-real-time ingest suits tickets and incident channels. Streaming requires idempotent chunk IDs so updates tombstone old chunks. Mixing batch and streaming into one index without version tags confuses “as of when” answers—especially in ops copilots.

Handoff to generation team

When recall@10 meets SLO, document for prompt engineers: typical top-3 chunk token counts, metadata fields available, citation format, and forbidden behaviors (quoting boilerplate footers). Generation tuning without this handoff reproduces “model ignores context” tickets that are actually assembly bugs—chunks arrive without titles or in random order.

Include negative examples: queries where abstention is correct (no chunk above score threshold). Prompts should encourage abstention rather than synthesizing from weak matches—critical before regulated deployments (vertical RAG).

Anti-patterns catalog

• Character-based chunking on UTF-8 multibyte text—breaks CJK and emoji-dense tickets.
• Re-embedding unchanged chunks after nightly full rebuilds—wastes GPU without recall gain.
• Single global k for FAQ and analyst workflows—tune k per route.
• Ignoring table captions—captions often carry the metric definition users query.
• Mixing draft and published docs without environment metadata—answers cite unreleased policy.

Review this catalog in quarterly ingest retros—teams reintroduce anti-patterns when new parsers ship under deadline pressure.

Finally, align chunking milestones with eval milestones in the same sprint review—shipping a new parser without recall@10 comparison is how regressions reach production during doc platform migrations.

Document parser version in every chunk metadata field—when legal asks why an answer changed, “we upgraded PDF extraction v3→v4” is an actionable answer; “the model changed” is not.

Chunking is unglamorous infrastructure—teams that fund only generator upgrades learn this during the first enterprise doc migration. Treat parser and chunker ownership as first-class platform roles, not one-off ingest scripts.

Sources

1. MTEB Embedding Leaderboard — public embedder comparisons (task mix ≠ your corpus).
2. Sentence Transformers documentation — training and inference patterns for open embedders.
3. Lewis et al., RAG (2020) — retrieval-generation baseline architecture.

What we did not test: We did not publish new MTEB submissions or private embedder rankings for this guide. Model names and leaderboard positions change; validate on your gold set before production cutover.

Corrections: Update embedder examples and dimension notes when index versions change; revise as-of date at top.

Next step

If entity relationships dominate your queries, read GraphRAG and knowledge graphs for enterprises. Otherwise instrument metrics with evaluate retrieval: recall and faithfulness.