Blog

RAG Pipeline: Architecture, Steps and Real Numbers

The 9 steps of a RAG pipeline, from loading docs to the final answer, with a framework-free build, measured costs, and the bug each step can cause.

9 min read

A RAG pipeline is two pipelines. One runs once, ahead of time: it cuts your documents into chunks, turns each chunk into an embedding, and stores the vectors. The other runs on every question: it embeds the question, finds the closest chunks, pastes them into the prompt, and asks the model.

That’s the whole architecture. This post walks through every step on a real bot, DocBot, a support assistant for a made-up SaaS company with 24 help docs. Every number below comes from DocBot’s runs, not from a vendor slide.

RAG architecture in one diagram

INDEXING  (runs once, and again when docs change)

  docs ──► 1 load + clean ──► 2 chunk ──► 3 embed ──► 4 store
  24 .md      plain text        67 chunks    67 × 1,536     vectors + ids
                                             numbers        + metadata

QUERY  (runs on every question)

  question ──► 5 embed ──► 6 retrieve ──► 7 filter / rerank ──► 8 augment ──► 9 generate
               1 × 1,536    top 4 by        drop stale or        one message:   answer
               numbers      cosine          off-limits chunks    rules + slips  + slip ids
                                                                 + question

Steps 1 to 4 decide what can be found. Steps 5 to 9 decide what the model reads. Most RAG bugs live in the first half and show up in the second.

Indexing: steps 1 to 4

1. Load and clean the documents

Read every source into plain text: markdown, PDFs, HTML pages, ticket exports. Strip navigation, cookie banners and repeated footers. They match a lot of questions and help none.

This is also where you decide what goes in. DocBot’s corpus has a 2025 refund policy (30 days) sitting next to the 2026 one (15 days). Both were loaded. That one decision causes the biggest bug in this post, at step 6.

2. Chunk: cut documents into slips

The model gets a few chunks per question, never the whole corpus. So each chunk should answer one thing on its own.

DocBot’s rule is simple: one chunk per ## section, and any section over 200 tokens is split at blank lines. Result: 24 docs → 67 chunks, median 69 tokens, smallest 15, largest 195. Each chunk keeps an id like refund_policy_2026.md#2 (file name + section number), which the model cites later.

Section-based chunking works when your docs have real headings. When they don’t (scanned PDFs, chat logs), the common fallback is fixed-size windows of a few hundred tokens with some overlap. There is no universal best chunk size. Test it on your own questions.

3. Embed every chunk

An embedding model turns each chunk into a list of numbers where similar meaning ends up close together. DocBot uses text-embedding-3-small: 1,536 numbers per chunk.

The cost is small. All 67 chunks total 5,156 tokens. At $0.02 per million tokens, embedding the whole corpus costs about $0.0001. You pay it again only for chunks that change.

4. Store the vectors

Store each vector with its chunk text, its id, and any metadata you’ll want to filter on later: document date, version, product, who may read it.

Do you need a vector database? Not at this size. Searching DocBot’s 67 vectors with plain numpy takes 4.7 µs per question. We also timed brute force on 1 million generated vectors: 28 ms per query with numpy, against 0.3 ms with an HNSW index (which still found 9,980 of the true top 10,000 neighbours). An index starts to matter somewhere past a hundred thousand chunks, or when you need filters and updates handled for you.

Query: steps 5 to 9

5. Embed the question

Same embedding model as step 3, always. Vectors from two different models live on different maps, so their distances mean nothing.

Some models also want a prefix: the e5 family expects query: on questions and passage: on chunks. Read the model card.

6. Retrieve the top k chunks

Compare the question vector with every chunk vector (cosine similarity) and keep the best k. DocBot uses k = 4.

Here is what came back for “How many days do I have to ask for a refund?”:

Rank Chunk Cosine
1 refund_policy_2025.md#2 (old policy: 30 days) 0.5650
2 refund_policy_2026.md#2 (current: 15 days) 0.5570
3 refund_policy_2026.md#5 (how to request) 0.4952
4 refund_policy_2025.md#5 (how to request) 0.4890

The stale policy ranked first. Retrieval did its job: both chunks really are about the refund window. The bug was loading the old document in step 1.

7. Filter and rerank (optional, often needed)

Two cheap fixes sit between retrieval and the prompt:

  • Metadata filters. Keep only version = current, or only docs this user may read. That one filter would have removed the 2025 chunk above. Filter inside the search when your store supports it: filtering after taking the top 4 can leave you with fewer than 4 chunks.
  • Reranking. Fetch more candidates (say 20), then let a cross-encoder reranker score each one against the question and keep the best 4. It’s slower than a vector lookup, but it reads question and chunk together, so it catches near-misses that cosine ranks wrong.

8. Augment: build one message

The model has exactly one input. DocBot’s message has three parts: instructions, the 4 chunks with their ids, and the question.

SYSTEM:
You are DocBot, the support assistant for Tarnik. Answer the customer briefly,
using only the slips below. Cite the slip id you used in square brackets,
like [file.md#2]. If the slips don't contain the answer, say you don't know.

USER:
Slips:

[refund_policy_2025.md#2]
## 2. Refund window
You can request a full refund within 30 days of your purchase or renewal date. ...

[refund_policy_2026.md#2]
## 2. Refund window
You can request a full refund within 15 days of your purchase or renewal date. ...

(2 more slips)

Question: How many days do I have to ask for a refund?

That message is 438 tokens. Pasting all 24 docs instead would be 5,194 tokens, about 12× more, on every question.

9. Generate

Send the message at temperature 0. DocBot answered:

You have 15 days from your purchase or renewal date to request a full refund [refund_policy_2026.md#2].

It then added a line saying an older policy allowed 30 days. The slip ids are how we traced that line back to the stale chunk in seconds. Always ask for citations.

One more trap here: our first call used max_tokens: 300 and the answer was cut off mid-sentence. The model had spent 181 of those 300 tokens on hidden thinking. Leave room.

Build it without a framework

Here are the nine steps in about 40 lines. No LangChain, no vector database. Put your docs in a docs/ folder as markdown.

import re
from pathlib import Path
import numpy as np, tiktoken
from openai import OpenAI
from anthropic import Anthropic

openai, llm = OpenAI(), Anthropic()
enc = tiktoken.get_encoding("cl100k_base")
SYSTEM = ("Answer briefly, using only the slips below. Cite the slip id you used "
          "in square brackets. If the slips don't contain the answer, say you don't know.")

def chunk(doc_name, text, limit=200):
    """Step 2: one chunk per '## ' section; long sections split at blank lines."""
    out = []
    for n, section in enumerate(re.split(r"\n(?=## )", text.strip())):
        parts, cur = [], ""
        for para in section.split("\n\n"):
            cand = (cur + "\n\n" + para).strip()
            if cur and len(enc.encode(cand)) > limit:
                parts.append(cur); cur = para
            else:
                cur = cand
        parts.append(cur)
        out += [{"id": f"{doc_name}#{n}" + ("" if j == 0 else chr(97 + j)), "text": p}
                for j, p in enumerate(parts)]
    return out

def embed(texts):
    """Steps 3 and 5: same model for chunks and questions, unit length."""
    r = openai.embeddings.create(model="text-embedding-3-small", input=texts)
    v = np.array([d.embedding for d in r.data])
    return v / np.linalg.norm(v, axis=1, keepdims=True)

# Indexing (steps 1-4): load, chunk, embed, keep in memory
chunks = [c for p in sorted(Path("docs").glob("*.md")) for c in chunk(p.name, p.read_text())]
vecs = embed([c["text"] for c in chunks])

def answer(question, k=4):
    q = embed([question])[0]                                  # 5. embed the question
    top = np.argsort(-(vecs @ q))[:k]                         # 6. retrieve top k
    slips = "\n\n".join(f"[{chunks[i]['id']}]\n{chunks[i]['text']}" for i in top)
    msg = llm.messages.create(                                # 8 + 9. augment, generate
        model="claude-haiku-5-5", max_tokens=1024, temperature=0, system=SYSTEM,
        messages=[{"role": "user", "content": f"Slips:\n\n{slips}\n\nQuestion: {question}"}])
    return "".join(b.text for b in msg.content if b.type == "text")

print(answer("How many days do I have to ask for a refund?"))
import { readdirSync, readFileSync } from "node:fs";
import OpenAI from "openai";
import Anthropic from "@anthropic-ai/sdk";
import { getEncoding } from "js-tiktoken";

const openai = new OpenAI();
const llm = new Anthropic();
const enc = getEncoding("cl100k_base");
const SYSTEM = "Answer briefly, using only the slips below. Cite the slip id you used " +
  "in square brackets. If the slips don't contain the answer, say you don't know.";

// Step 2: one chunk per '## ' section; long sections split at blank lines
function chunk(docName, text, limit = 200) {
  const out = [];
  text.trim().split(/\n(?=## )/).forEach((section, n) => {
    const parts = [];
    let cur = "";
    for (const para of section.split("\n\n")) {
      const cand = (cur + "\n\n" + para).trim();
      if (cur && enc.encode(cand).length > limit) { parts.push(cur); cur = para; }
      else cur = cand;
    }
    parts.push(cur);
    parts.forEach((p, j) =>
      out.push({ id: `${docName}#${n}${j ? String.fromCharCode(97 + j) : ""}`, text: p }));
  });
  return out;
}

// Steps 3 and 5: same model for chunks and questions, unit length
async function embed(texts) {
  const r = await openai.embeddings.create({ model: "text-embedding-3-small", input: texts });
  return r.data.map(({ embedding: e }) => {
    const n = Math.hypot(...e);
    return e.map((x) => x / n);
  });
}
const dot = (a, b) => a.reduce((s, x, i) => s + x * b[i], 0);

// Indexing (steps 1-4): load, chunk, embed, keep in memory
const chunks = readdirSync("docs").filter((f) => f.endsWith(".md")).sort()
  .flatMap((f) => chunk(f, readFileSync(`docs/${f}`, "utf8")));
const vecs = await embed(chunks.map((c) => c.text));

async function answer(question, k = 4) {
  const [q] = await embed([question]);                       // 5. embed the question
  const top = vecs.map((v, i) => ({ i, s: dot(v, q) }))      // 6. retrieve top k
    .sort((a, b) => b.s - a.s).slice(0, k);
  const slips = top.map(({ i }) => `[${chunks[i].id}]\n${chunks[i].text}`).join("\n\n");
  const msg = await llm.messages.create({                    // 8 + 9. augment, generate
    model: "claude-haiku-5-5", max_tokens: 1024, temperature: 0, system: SYSTEM,
    messages: [{ role: "user", content: `Slips:\n\n${slips}\n\nQuestion: ${question}` }],
  });
  return msg.content.filter((b) => b.type === "text").map((b) => b.text).join("");
}

console.log(await answer("How many days do I have to ask for a refund?"));

Step 7 is missing on purpose: add a metadata filter once you see stale or off-limits chunks in your top 4.

Starting defaults

Setting DocBot’s value Why
Chunking One per ## section, split over 200 tokens Help docs already have one topic per section
Chunk id file.md#section Lets the model cite, and you debug
Embedding model text-embedding-3-small, 1,536 dims Cheap, good on English. Weak on Hinglish (see below)
k 4 Enough for one fact plus context; 438-token message
Temperature 0 Same question, same answer
max_tokens 1,024 300 got cut off by hidden thinking
Vector store numpy array in memory 67 chunks search in 4.7 µs

These are starting points measured on one small corpus. Change one setting at a time and re-score on the same questions.

What breaks at each step

Every one of these happened on DocBot.

Symptom Step What we measured Fix
Answer mentions an old policy 1, 7 Stale 2025 chunk ranked #1 (0.5650) Remove old versions, or filter on version
Hinglish question finds the wrong things 3 “paisa wapas kab milega?” vs “How long does a refund take?”: cosine 0.19 with text-embedding-3-small, 0.72 with gemini-embedding-001 Use a multilingual embedding model
Exact ID lookups fail 3, 6 Searching “INV-2041”: the right invoice ranked 10th of 10. INV-2041 and INV-2047 scored 0.98 alike Add keyword search next to vectors (hybrid search)
Answer stops mid-sentence 9 181 of 300 output tokens went to thinking Raise max_tokens
Wrong answer, confident tone 6 The model only knows what retrieval hands it Check the cited slip ids first
Chunk makes no sense alone 2 Chunks carry no doc title, so the year lives only in the id Prepend the doc title to each chunk

The last column is the order to debug in: look at the slips before you blame the model.

What a RAG pipeline costs

DocBot’s measured refund-window call: 560 input tokens and 223 output tokens (97 of them thinking), on Claude Haiku at $0.10 per million input and $0.50 per million output. That’s $0.00017 per question, or about $0.17 per 1,000 questions. Embedding the question adds about $0.0000002.

Approach Input tokens per question (cl100k) Input cost per 1,000 questions
Paste all 24 docs 5,194 $0.52
RAG, top 4 chunks 438 $0.044

At 24 docs, both are cheap. What scales is the ratio: double the docs and “paste everything” doubles, while RAG stays at 4 chunks.

How to know your pipeline works

Write down real questions with known answers (we use 50 for DocBot) and score two things separately:

  1. Retrieval: is the chunk that holds the answer in the top k? This is recall@k, and it needs no LLM to measure.
  2. Answer: is the answer correct, and supported by the slips it cites?

If retrieval fails, no prompt will save the answer. If retrieval passes and the answer is still wrong, then look at the prompt and the model.

When you don’t need a RAG pipeline

If all your docs fit comfortably in the context window (a few hundred pages), paste them in and turn on prompt caching so repeat questions don’t pay full price for the same text. RAG earns its place when the corpus is too big, changes often, or needs per-user permissions.

FAQ

What are the steps of a RAG pipeline?

Indexing: load, chunk, embed, store. Query: embed the question, retrieve the top k chunks, filter or rerank, build the prompt, generate. Some guides merge these into three words: retrieve, augment, generate.

Is a RAG pipeline the same as RAG architecture?

Mostly. “Pipeline” usually means the ordered steps; “architecture” adds the parts that run them: the embedding model, the vector store, the reranker, the LLM, and how indexing stays in sync with your docs.

Do I need LangChain or LlamaIndex?

No. The code above covers the whole pipeline. Frameworks help when you want loaders for many file types, ready-made rerankers, or agents. Learn the steps first, so you can tell which part of the framework is failing.

Do I need a vector database?

Not for thousands of chunks: a numpy array is fast enough. You want one when you reach hundreds of thousands of chunks, need filtering during search, or need to update vectors without rebuilding everything.

What chunk size should I use?

Start with your documents’ own structure (sections, FAQ entries). If there is none, try a few hundred tokens with a little overlap, then measure recall@k on your own questions and adjust.

Go deeper

Sources

  • Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020. arXiv:2005.11401
  • Malkov and Yashunin, Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs, 2016. arXiv:1603.09320
  • DocBot measurements: our own runs, November 2026 (embedding text-embedding-3-small and gemini-embedding-001, answers from Claude Haiku, token counts with cl100k_base).