RAG Pipeline Architecture: End-to-End System Design Explained

1. Introduction

In the previous article, we defined RAG (Retrieval-Augmented Generation) as an architecture that grounds LLM responses in external knowledge. But RAG is not a single technique or a simple API call. In production, it’s a multi-stage pipeline system—a carefully orchestrated sequence of offline and online processes that together transform raw documents into accurate, cited answers.

Understanding this pipeline architecture is what separates a proof-of-concept chatbot from a reliable, scalable production system. Every stage—ingestion, chunking, embedding, indexing, retrieval, ranking, context assembly, and generation—carries design choices that cascade downstream. A poor chunking decision made during ingestion will silently degrade retrieval quality weeks later. An undersized vector database will buckle under production load.

This article walks you through the complete RAG pipeline, stage by stage, explaining what each component does, how it connects to the others, where the bottlenecks hide, and what production-grade patterns look like. Let's trace the full data flow.

2. High-Level RAG Pipeline Overview

The RAG pipeline is often split into two major phases: an offline ingestion phase and an online query phase. The following diagram shows the complete end-to-end flow:

Figure 1: The complete RAG pipeline. The offline phase (top) processes documents ahead of time. The online phase (bottom) handles live queries, using the pre-built vector index.

Every stage deserves close attention.

3. Stage 1: Data Ingestion

The pipeline begins with data sources—the raw material that gives your RAG system its knowledge. Common sources include:

• PDF documents (manuals, contracts, research papers).
• Web pages (internal wikis, documentation portals, scraped public sites).
• APIs (live data feeds, CRM records, database exports).
• Databases (structured records that need text extraction).

During ingestion, data must be cleaned and normalized:

• Strip headers, footers, and navigation boilerplate.
• Remove corrupted or non-text content (images, scanned pages without OCR).
• Normalize Unicode, whitespace, and date formats.
• Extract text from proprietary formats (.docx, .pptx, HTML).

This stage is unglamorous but critical. Garbage entering the pipeline—duplicate content, malformed text, irrelevant markup—directly pollutes retrieval quality downstream. Many teams underestimate the engineering effort needed here.

4. Stage 2: Chunking Strategy

An entire 50-page document cannot be embedded as a single vector and retrieved precisely. It must be split into chunks—smaller, coherent segments that each receive their own embedding.

Chunking involves critical trade-offs:

Chunk Size	Advantages	Disadvantages
Small (128–256 tokens)	Highly precise retrieval, fast embedding.	May lack context; fragments sentences; increases total chunk count.
Medium (512–1024 tokens)	Good balance of context and precision.	Requires careful overlap tuning.
Large (2048+ tokens)	Rich context, fewer chunks to manage.	Embedding may be diluted; retrieval less precise; hits context window faster.

Overlap is essential at medium/large sizes. A 10–20% token overlap between consecutive chunks prevents critical sentences from being cut at chunk boundaries. For example, an overlap ensures that “The contract shall terminate...” isn't separated from “...upon 30 days written notice.”

Chunking can be naive (fixed character count) or smart (splitting on sentence boundaries, paragraphs, or even semantic breaks using a small model). The choice profoundly affects retrieval quality.

Deeper dive: Our dedicated article on Chunking Strategies in RAG covers semantic splitting, recursive strategies, and domain-specific approaches.

5. Stage 3: Embedding Generation

Once documents are chunked, each chunk must be converted into a dense vector using an embedding model. This vector is the semantic fingerprint of the chunk—chunks with similar meanings will have vectors that are close together in high-dimensional space.

Embedding generation is a batch process in the offline phase:

1. All chunks are collected.
2. An embedding model (e.g., text-embedding-3-small, BGE-large, E5-mistral) processes them, typically in parallel.
3. Each chunk receives a fixed-length vector (e.g., 1024 or 4096 dimensions).
4. The vector is paired with metadata (original text, source, date, chunk index) for storage.

The choice of embedding model locks in your retrieval behavior. Switching later means re-embedding everything—a costly migration. Models optimized for retrieval (like those on the MTEB leaderboard) use contrastive learning to pull similar documents together and push dissimilar ones apart.

Embedding model selection: Our Embedding Models for RAG Systems article benchmarks popular choices and explains trade-offs between quality, speed, and cost.

6. Stage 4: Vector Database Storage

The embedded chunks—vectors plus their metadata—are stored in a vector database. This specialized database indexes vectors using approximate nearest neighbor (ANN) algorithms, enabling similarity search over millions of vectors in milliseconds.

Key capabilities of a vector database in RAG:

• Similarity search: Given a query vector, return the top‑k most similar document vectors.
• Metadata filtering: Narrow results by source, date, author, or any other tag before or after vector search.
• Scalability: Horizontal scaling to handle billions of vectors.
• CRUD operations: Update, delete, or insert vectors as documents change.

Popular choices for RAG pipelines:

Database	Deployment	Distinguishing Feature
Pinecone	Managed cloud	Zero-ops, serverless scaling
Weaviate	Self-hosted / Cloud	Built-in hybrid search, GraphQL
Milvus	Self-hosted / Cloud	High throughput, cloud-native
Qdrant	Self-hosted / Cloud	Fast Rust core, rich filtering
pgvector	PostgreSQL extension	Familiar SQL, no new infrastructure

The vector database is the runtime heart of the RAG system—any latency here directly impacts user experience.

Vector DB deep dive: Vector Database Explained covers indexing algorithms (HNSW, IVF), memory layouts, and production deployment patterns.

7. Stage 5: Query Processing

When a user submits a query, the online phase begins. The first step is to normalize and embed the query.

• Normalization: The raw query string may be trimmed, lowercased, or processed to remove irrelevant characters.
• Query rewriting (optional): In advanced systems, the user’s question may be rephrased or decomposed into sub-queries to improve retrieval. This can be as simple as using the LLM to expand the query with synonyms.
• Embedding: The normalized query is passed through the same embedding model used during ingestion. Consistency here is non-negotiable: a mismatched embedding model produces vectors in a completely different semantic space, making retrieval useless.

The output is a single dense vector representing the user’s intent.

8. Stage 6: Retrieval Layer

The query vector is sent to the vector database, which performs a similarity search. It compares the query vector against all stored document chunk vectors, computes a similarity score (cosine similarity, dot product, or Euclidean distance), and returns the top‑k most similar chunks.

• k is a tunable parameter. A low k (3–5) minimizes noise but risks missing relevant information. A high k (20–50) improves recall but consumes more context window tokens and may introduce irrelevant results.
• Most vector databases support metadata filtering at this stage. For example, you can restrict the search to documents from a specific department or published after a certain date, dramatically improving precision.

The output of this stage is a ranked list of candidate chunks, each with a similarity score.

9. Stage 7: Ranking and Re-ranking

Initial vector similarity is a decent first pass but often insufficient for high-precision applications. A chunk might score highly on vector similarity because it shares vocabulary or topic with the query, but fail to actually contain the answer.

Re-ranking is a second-pass scoring step that uses a more sophisticated (and slower) model to re-order the top‑k candidates. Common approaches:

• Cross-encoder models: Instead of comparing embeddings, a cross-encoder takes the query and a candidate chunk together as input and outputs a relevance score. Models like BGE-reranker or Cohere Rerank are purpose-built for this.
• LLM-based re-ranking: For small candidate sets, the LLM itself can score or select the most relevant chunks given the query.

Re-ranking improves precision dramatically—often boosting answer accuracy by 10–20%. The typical pattern: retrieve 50–100 candidates with vector search, then re-rank the top 50 down to the final 5–10 chunks sent to the LLM.

Re-ranking deep dive: Reranking in RAG Systems covers cross-encoder architecture, latency trade-offs, and multi-stage ranking pipelines.

10. Stage 8: Context Assembly

The final set of top chunks must now be assembled into a coherent prompt for the LLM. This is context assembly—and it’s easy to get wrong.

Key considerations:

• Token budget: The context window is finite. You must allocate tokens between the system prompt, the retrieved chunks, the user query, and the expected output length. A common breakdown: 10% system prompt, 70% retrieved context, 20% for user query and output headroom.
• Ordering: Chunks should be ordered in a way that helps the LLM reason. Options include: by relevance score (most relevant first), by source document order (for narrative coherence), or by chunk index (to reconstruct the original flow).
• Instructions: The prompt should explicitly tell the LLM: “Answer based only on the provided context. If the context doesn’t contain the answer, say so.” This grounds the response and reduces hallucination.
• Citation formatting: Many production systems include source metadata (document title, page number) in the prompt and instruct the model to cite its sources.

Poor assembly can turn excellent retrieval into a confusing or unhelpful answer.

Context management: See our Context Window article for strategies on managing token limits across long conversations.

11. Stage 9: LLM Generation

The augmented prompt—system instructions + retrieved context + user query—is sent to the LLM. The model reads the provided context, reasons over it, and generates a natural language response.

Because the necessary facts are embedded in the prompt, the model relies far less on its internal knowledge. This is what enables accurate answers about proprietary policies, recent events, or niche domains.

The generation stage inherits all the standard considerations of LLM inference:

• Sampling parameters (temperature, top‑p) control creativity.
• Streaming delivers tokens to the user in real time.
• The model stops at an end-of-sequence token or a configured max length.

LLM inference deep dive: How LLM Inference Works covers the prefill and decode phases, KV cache, and serving infrastructure.

12. Online vs Offline Pipeline

RAG systems have two distinct operational modes:

Aspect	Offline (Ingestion)	Online (Query)
When it runs	Periodically (batch) or on document update	Real-time, per user request
Throughput	High (batch embedding, indexing)	Low per user, high aggregate
Latency sensitivity	Low (minutes to hours acceptable)	Critical (sub-second expected)
Primary cost	Embedding compute, vector storage	Vector search, LLM inference
Scalability axis	Document volume	Concurrent users

Understanding this split is essential for capacity planning. You can scale offline ingestion independently from online serving, and you can optimize each phase for its specific constraints.

13. Latency Breakdown in RAG Systems

When a user submits a query, the total latency adds up across multiple stages:

Stage	Typical Latency	Notes
Query embedding	10–50 ms	Depends on model size and API
Vector search	5–50 ms	Depends on index size, algorithm, filtering
Re-ranking	50–200 ms	Cross-encoder models are heavier
Context assembly	< 5 ms	String concatenation, negligible
LLM generation (TTFT)	200–1000 ms	Varies with model size and prompt length
LLM generation (per token)	10–30 ms	Streaming improves perceived latency

Total end-to-end latency typically ranges from 500 ms to 2 seconds. Optimizations like caching frequent queries, pre-computing embeddings for common questions, and using faster re-rankers can shrink these numbers considerably.

14. Common Bottlenecks

In production, bottlenecks usually hide in one of these areas:

• Poor chunking: Irrelevant or orphaned chunks pollute retrieval results, forcing the LLM to sift through noise.
• Weak embeddings: A generic embedding model that doesn’t capture domain-specific semantics will miss relevant documents.
• Retrieval noise: Too many retrieved chunks (high k) dilutes the useful context and wastes token budget.
• Context overflow: Exceeding the context window causes truncation—often silently dropping the most relevant information that was at the end.
• Slow vector DB: An under-provisioned or poorly indexed database becomes the latency tail that drags down the whole pipeline.
• Missing re-ranking: Without re-ranking, the most relevant chunk might be buried at position 17 in the results, never reaching the LLM.

Each bottleneck has a targeted solution, which we’ll explore throughout this RAG series.

15. Production Architecture Patterns

As RAG systems mature, several architectural patterns emerge:

• Single-stage RAG: The simplest form: embed, search, top‑k, generate. Suitable for small document sets and low-complexity queries.
• Multi-stage RAG: Adds re-ranking and possibly query rewriting or decomposition. The standard for high-quality production systems.
• Hybrid RAG: Combines dense (vector) retrieval with sparse (keyword, BM25) retrieval. The results from both paths are merged (fusion), capturing both semantic meaning and exact keyword matches.
• Cached retrieval: Frequently asked queries and their retrieved contexts are cached to skip embedding and search entirely, reducing latency and cost.
• Agentic RAG: The LLM is given the ability to iteratively query the vector database, refine the search, or request specific documents—turning retrieval into a multi-turn reasoning loop.

Choosing the right pattern depends on query complexity, document volume, latency budget, and accuracy requirements.

16. Real-World System Example

Consider an enterprise knowledge assistant for a large company:

• Ingestion: 50,000 internal documents (PDFs, Confluence pages, Google Docs) are cleaned, chunked to 512 tokens with 10% overlap, embedded with BGE-large-en, and stored in a Milvus instance.
• Querying: An employee asks, “What is the parental leave policy for the Berlin office?”
• Pipeline execution:
  1. Query embedding (10 ms).
  2. Vector search in Milvus with metadata filter location=Berlin AND topic=HR (15 ms).
  3. Top-20 candidates re-ranked by a cross-encoder, yielding top-5 (80 ms).
  4. Context assembled: chunks from the official policy document and an FAQ page, with source citations.
  5. Prompt sent to Llama 3 70B, which generates a cited response (600 ms).
• Result: A precise, sourced answer—delivered in under 1 second—that would be impossible for a standalone LLM.

This architecture handles thousands of queries per day, updates its knowledge base nightly, and can be audited by reviewing retrieved chunks.

17. Relationship to Other RAG Components

The pipeline connects all the major RAG and LLM concepts:

Each link in this map is a dedicated topic. The pipeline provides the scaffolding; the components provide the capability. Mastering both the overall flow and the individual pieces is what makes a proficient RAG engineer.

18. Key Takeaways

• RAG is a full pipeline system, not a single step. It spans offline ingestion (cleaning, chunking, embedding, indexing) and online query (embedding, search, re-rank, assemble, generate).
• Every stage matters. A poor chunking decision made during ingestion will degrade retrieval quality indefinitely. The pipeline is only as strong as its weakest link.
• Ingestion is an engineering-heavy phase. Data cleaning, normalization, and chunking require careful design and ongoing maintenance.
• Retrieval quality depends on embedding model choice, chunk strategy, and vector database performance.
• Re-ranking dramatically improves precision at a modest latency cost, and is a hallmark of production-grade systems.
• Context assembly is a delicate balancing act between providing enough information to ground the LLM and staying within the token budget.
• The online/offline split allows independent scaling and optimization of ingestion throughput vs. query latency.
• Production RAG systems use patterns like hybrid search, caching, and agentic retrieval to meet real-world demands.

Understanding the RAG pipeline architecture equips you to design, debug, and scale knowledge-grounded AI applications. In the next articles, we’ll zoom into each stage—starting with the vector database that sits at the pipeline’s core.