Why I built it myself
Frameworks make RAG easy to ship and hard to understand. A support bot gives a customer the wrong plan price, and nobody can say whether the problem was the documents, the chunks, the search or the prompt. I wanted to own each of those decisions myself.
Chunk size, overlap and top-k are set once and never questioned, yet they decide what the model ever sees.
When the answer is wrong, teams change the prompt because it's the only thing they can see.
Without citations, users and stakeholders can't check where an answer came from.
PMs need to reason about recall, precision and cost without reading the code.
- Needs
- To understand which knob caused a bad answer, and to defend a retrieval choice with evidence.
Why did it say SSO is on the free plan?
- Needs
- Answers about pricing and policy that are right every time, with a source to point to.
One wrong price quote is a refund and a lost customer.
- Needs
- A fast way to compare settings side by side, and to export chunks and scores for debugging.
Show me the scores, not the vibes.
The six stages, and the question I asked at each

Stages 1–2: documents in, chunks out
I used a realistic SaaS knowledge base: three pricing plans (Starter, Pro, Business), a cancellation policy and a data-retention policy. These are the questions support teams get every day, and wrong answers cost money.
Chunking is fully exposed. You pick a strategy, chunk size and overlap, and see every chunk with its word count and embedding status. With 100 words and 20 of overlap, five documents became three chunks of 100, 83 and 3 words. That turned out to matter a lot.

Stages 3–4: embed and retrieve
Chunks are embedded with OpenAI's text-embedding-3-small and kept in a small vector store. A query is ranked against them by cosine similarity, and the top-k chunks are shown with their similarity scores.
Showing the score was a deliberate choice. It's the number that tells you whether retrieval actually found something relevant or just returned whatever was least bad.

Stages 5–6: prompt and cited answer
The composed prompt is shown in full: the system instruction, each retrieved chunk with its score, and the user's question. Nothing is hidden.
The system rule is simple: answer from the context and always cite chunk IDs. Every sentence in the answer carries a clickable citation, so you can trace each claim back to the text it came from.

The side panel: speed, shape and comparison
- Performance timeline: time per stage, so latency has a clear owner.
- Embedding visualization: chunks projected to 2D, so you can see whether they actually separate by topic.
- Export: chunks, embeddings and results as files for debugging or sharing.
- Comparison mode: run a "Conservative" setting (top-k 3, minimum similarity 70%) against a "Balanced" one (top-k 5, 50%) side by side.
I asked it real customer questions
For each question I wrote down the right answer from the source documents first, then compared it with what the pipeline returned and traced the gap back to a stage.
Business, at $49 per seat per month, with a minimum of 3 seats. Starter and Pro don't include it.
One 100-word chunk held Starter, Pro and Business together. The answer put Starter's name next to Business's features, so a reader could think SSO is free.
$147 per month: 3 seats minimum × $49 per seat.
A 3-word leftover chunk ("workspace access permissions.") ranked first. The price sat in another chunk, and nothing asked the model to multiply seats by price.

Reading the scores
For question 2, the ranked context was:
| Rank | Chunk | Words | Similarity |
|---|---|---|---|
| 1 | chunk_003 · "workspace access permissions." | 3 | 0.225 |
| 2 | chunk_001 · plans and prices | 100 | 0.223 |
| 3 | chunk_002 · Business, policies | 83 | low |
All the scores sat between about 0.01 and 0.23. When chunks mix several topics, their vectors become an average of everything, and nothing stands out. That's also why comparison mode returned zero chunks at 50% and 70% minimum similarity: those thresholds never fit this score range.
Try the chunking yourself
This uses the same five documents. Move the sliders and watch how fixed-size chunks blend plans together, then switch to document-aware chunking.
What I'd fix, in order
Three plans in one chunk let the answer attach Business features to the Starter plan.
FixSplit on document boundaries first, then by size. Attach plan name and doc ID as metadata to every chunk.
A 3-word leftover ranked first for a pricing question.
FixSet a minimum chunk size and merge tails into the previous chunk.
50% and 70% cut-offs returned nothing, because real scores sat below 25%.
FixCalibrate thresholds from the score distribution on a test set, and answer "I don't know" below them.
"SSO" is a precise keyword that embeddings blur.
FixAdd keyword search (BM25) alongside vectors, then rerank the combined results.
The facts for "$49 × 3 seats" were retrieved, but never combined.
FixKeep prices as structured data the model can calculate from, and tell the prompt to show its working for cost questions.
Because every sentence cited its chunk, I could trace each wrong answer to its cause in minutes. Without citations, both errors would have looked like a model problem.
What I'm measuring next
Two hand-checked questions found two real bugs. The next step is a golden set of 50 real customer questions, each with the correct answer and source document, scored on every settings change.
Right answer, from the right document, on the golden question set.
- Target > 95%
- Hit rate at top 3 > 90%
- Chunks mixing documents 0
- Similarity gap, best vs. next tracked
- Wrong price or plan quoted 0
- "I don't know" when unsupported > 90%
- p95 end-to-end latency < 2s
- Cost per answered question tracked
What I learned
How you split a document decides what the model can ever know. It deserves the same scrutiny as a pricing page.
Without an expected answer, "sounds plausible" passes for correct. Both bugs only showed up because I knew what the answer should be.
Scores, prompts and citations on screen turn RAG from a black box into something a whole team can reason about.