AI project · AI infrastructure · Built from scratch

RAG from Scratch

I wanted to see every step.A RAG pipeline visualizer that shows each stage, from document to cited answer.

Most PMs ship RAG features without seeing what's inside them. I built the whole pipeline myself (chunking, embeddings, retrieval, prompt, answer), pointed it at a set of SaaS pricing documents, and asked it the questions real customers ask. Then I studied exactly where it went wrong.

go play with the chunking sliders ↓

RoleBuilder, PM
EmbeddingsOpenAI text-embedding-3-small
RetrievalCosine similarity, top-k
StatusWorking prototype
Document input and chunking settings with chunk size 100 and overlap 20
Prompt construction and an LLM answer with chunk citations
ProblemRAG is a black box

Teams tune chunk size, top-k and thresholds by guesswork, and can't explain wrong answers.

BetMake every step visible

If you can see what each stage produced, you can tell where an answer went wrong.

BuiltA 6-stage visualizer

Input, chunking, embedding, retrieval, prompt and a cited answer, with timing and comparison mode.

FoundChunking is the product

Most wrong answers traced back to how documents were split, not to the model.

01Why I built it from scratch

Why I built it myself

Frameworks make RAG easy to ship and hard to understand. A support bot gives a customer the wrong plan price, and nobody can say whether the problem was the documents, the chunks, the search or the prompt. I wanted to own each of those decisions myself.

Hidden defaults

Chunk size, overlap and top-k are set once and never questioned, yet they decide what the model ever sees.

No way to debug

When the answer is wrong, teams change the prompt because it's the only thing they can see.

Untrusted answers

Without citations, users and stakeholders can't check where an answer came from.

A gap between PM and engineering

PMs need to reason about recall, precision and cost without reading the code.

P
The AI product manager
Primary
Needs
To understand which knob caused a bad answer, and to defend a retrieval choice with evidence.
Why did it say SSO is on the free plan?
S
The support lead
Deploying a help bot
Needs
Answers about pricing and policy that are right every time, with a source to point to.
One wrong price quote is a refund and a lost customer.
E
The engineer
Tuning the pipeline
Needs
A fast way to compare settings side by side, and to export chunks and scores for debugging.
Show me the scores, not the vibes.
02My thought process

The six stages, and the question I asked at each

1Document inputWhat is the source of truth?Paste text or upload TXT, MD or PDF.5 pricing & policy docs, 163 words
2ChunkingHow big is one fact?Strategy, chunk size and overlap are all adjustable.Fixed size · 100 words · 20 overlap
3EmbeddingHow do we represent meaning?Each chunk becomes a vector in a store.text-embedding-3-small
4RetrievalRecall or precision?Cosine similarity ranks chunks for the query.Top-k = 3
5PromptHow do we keep it grounded?System rule plus retrieved chunks plus the question."Always cite the chunk IDs"
6AnswerCan the user verify it?Every claim carries a citation chip.Citations per sentence
Stage 1 document input and stage 2 chunking

Stages 1–2: documents in, chunks out

I used a realistic SaaS knowledge base: three pricing plans (Starter, Pro, Business), a cancellation policy and a data-retention policy. These are the questions support teams get every day, and wrong answers cost money.

Chunking is fully exposed. You pick a strategy, chunk size and overlap, and see every chunk with its word count and embedding status. With 100 words and 20 of overlap, five documents became three chunks of 100, 83 and 3 words. That turned out to matter a lot.

Stage 3 embeddings and stage 4 retrieval with similarity scores

Stages 3–4: embed and retrieve

Chunks are embedded with OpenAI's text-embedding-3-small and kept in a small vector store. A query is ranked against them by cosine similarity, and the top-k chunks are shown with their similarity scores.

Showing the score was a deliberate choice. It's the number that tells you whether retrieval actually found something relevant or just returned whatever was least bad.

Stage 5 composed prompt and stage 6 answer with citations

Stages 5–6: prompt and cited answer

The composed prompt is shown in full: the system instruction, each retrieved chunk with its score, and the user's question. Nothing is hidden.

The system rule is simple: answer from the context and always cite chunk IDs. Every sentence in the answer carries a clickable citation, so you can trace each claim back to the text it came from.

Performance timeline, embedding visualization, export and comparison mode

The side panel: speed, shape and comparison

  • Performance timeline: time per stage, so latency has a clear owner.
  • Embedding visualization: chunks projected to 2D, so you can see whether they actually separate by topic.
  • Export: chunks, embeddings and results as files for debugging or sharing.
  • Comparison mode: run a "Conservative" setting (top-k 3, minimum similarity 70%) against a "Balanced" one (top-k 5, 50%) side by side.
113 mstotal pipeline
35 msembedding
39 msretrieval
39 msgeneration
03Answering product questions

I asked it real customer questions

For each question I wrote down the right answer from the source documents first, then compared it with what the pipeline returned and traced the gap back to a stage.

Question 1"Which plan supports SSO?"Partly right
Correct answer

Business, at $49 per seat per month, with a minimum of 3 seats. Starter and Pro don't include it.

What RAG said
"DOC-P01 | Starter Plan Price: $0 per month. [chunk_001] Includes 2,000 AI requests per user per day, shared team workspaces, role-based permissions, SSO… [chunk_001] DOC-P03 | Business Plan Price: $49 per seat…"
Why

One 100-word chunk held Starter, Pro and Business together. The answer put Starter's name next to Business's features, so a reader could think SSO is free.

Question 2"What is the minimum Business subscription cost?"Missed
Correct answer

$147 per month: 3 seats minimum × $49 per seat.

What RAG said
"Minimum 3 seats. [chunk_002] Business history: 365 days. [chunk_002] Business administrators can configure workspace access permissions. [chunk_002]"
Why

A 3-word leftover chunk ("workspace access permissions.") ranked first. The price sat in another chunk, and nothing asked the model to multiply seats by price.

Second run: retrieval, prompt and answer for the minimum Business cost question

Reading the scores

For question 2, the ranked context was:

RankChunkWordsSimilarity
1chunk_003 · "workspace access permissions."30.225
2chunk_001 · plans and prices1000.223
3chunk_002 · Business, policies83low

All the scores sat between about 0.01 and 0.23. When chunks mix several topics, their vectors become an average of everything, and nothing stands out. That's also why comparison mode returned zero chunks at 50% and 70% minimum similarity: those thresholds never fit this score range.

04Try it · chunking lab

Try the chunking yourself

This uses the same five documents. Move the sliders and watch how fixed-size chunks blend plans together, then switch to document-aware chunking.

Strategy
Chunks: 3Mixing several docs: 2Tiny tail chunks: 1Chunks containing "SSO": 2
P01 StarterP02 ProP03 BusinessP04 CancellationP05 Retention
05What I'd change

What I'd fix, in order

ProblemChunks cross document boundaries

Three plans in one chunk let the answer attach Business features to the Starter plan.

Fix

Split on document boundaries first, then by size. Attach plan name and doc ID as metadata to every chunk.

ProblemTiny tail chunks outrank real content

A 3-word leftover ranked first for a pricing question.

Fix

Set a minimum chunk size and merge tails into the previous chunk.

ProblemThresholds don't match the score range

50% and 70% cut-offs returned nothing, because real scores sat below 25%.

Fix

Calibrate thresholds from the score distribution on a test set, and answer "I don't know" below them.

ProblemExact terms get lost

"SSO" is a precise keyword that embeddings blur.

Fix

Add keyword search (BM25) alongside vectors, then rerank the combined results.

ProblemRetrieval isn't arithmetic

The facts for "$49 × 3 seats" were retrieved, but never combined.

Fix

Keep prices as structured data the model can calculate from, and tell the prompt to show its working for cost questions.

What workedCitations made the errors findable

Because every sentence cited its chunk, I could trace each wrong answer to its cause in minutes. Without citations, both errors would have looked like a model problem.

06How I'd measure it

What I'm measuring next

Two hand-checked questions found two real bugs. The next step is a golden set of 50 real customer questions, each with the correct answer and source document, scored on every settings change.

North starAnswers correct and correctly cited

Right answer, from the right document, on the golden question set.

  • Target > 95%
RetrievalDid we find the right text?
  • Hit rate at top 3 > 90%
  • Chunks mixing documents 0
  • Similarity gap, best vs. next tracked
GuardrailsIs it safe to show customers?
  • Wrong price or plan quoted 0
  • "I don't know" when unsupported > 90%
  • p95 end-to-end latency < 2s
  • Cost per answered question tracked
07What I learned

What I learned

Chunking is a product decision

How you split a document decides what the model can ever know. It deserves the same scrutiny as a pricing page.

Write the right answer first

Without an expected answer, "sounds plausible" passes for correct. Both bugs only showed up because I knew what the answer should be.

Visibility builds trust

Scores, prompts and citations on screen turn RAG from a black box into something a whole team can reason about.