Skip to content
All articles
AI Engineering

What Is RAG? A Practical Guide to Retrieval-Augmented Generation

RAG explained from the ground up: what embeddings and vector search actually do, how to build the write and read pipelines, when you don't need a document crawler — and the minimal FAQ-only RAG system running in production behind a support bot.

Rushit Jivani6 min read

The short version

  • RAG means fetching relevant text at question time and pasting it into the prompt, so the model answers from your content instead of its training data. It is two pipelines — a write path and a read path — meeting in a vector index.
  • Most teams over-build the write path. If your knowledge is short, curated entries (FAQs, policies), you can skip crawlers and chunkers entirely: one entry, one vector, one stable id.
  • The case study: a production support bot whose entire knowledge base is FAQ pairs in Pinecone — ids are hashes of the question, edits land from Slack, and a retrieval failure degrades to a cautious answer instead of an error.

Ask a language model about your product — your refund policy, your publishing flow — and it will answer fluently from nothing, because your product was not in its training data. Retrieval-augmented generation (RAG) is the standard fix, and despite the acronym it is a simple idea: before the model answers, fetch the relevant pieces of your own content and put them in the prompt. The model stops being an oracle and becomes a very good reader.

This guide builds the idea from the ground up — embeddings, vector search, the two pipelines — then walks through the deliberately minimal RAG system I run in production, which skips most of what RAG tutorials insist on, and explains why that was the right call.

What RAG is, mechanically

Three concepts, no magic:

Embeddings. An embedding model turns text into a vector — a long list of numbers — such that texts with similar meaning get nearby vectors. "Can I get my money back?" and "what's the refund policy?" share almost no words, but their vectors are close. That is the property everything else is built on.

Vector search. Store the vectors for your content in an index (Pinecone, pgvector, Weaviate…). At question time, embed the question with the same model and ask the index for the nearest stored vectors. Nearest vectors = most relevant passages, matched by meaning rather than keywords.

Augmentation. Take the top passages, paste them into the prompt with labels, and instruct the model: answer from this material; if the answer is not here, say so. The generation is "augmented" by retrieval — hence the name.

The whole system is two pipelines meeting in that index:

RAG is two pipelines that meet in an index

write path · when content changes

  1. Collect sourcesDocs, FAQs, policies — content a person vouches for.
  2. ChunkSplit into passages small enough to retrieve precisely. Short curated entries may already be chunks.
  3. EmbedOne embedding model turns each chunk into a vector. Pin this model — changing it means re-embedding everything.
  4. StoreVectors plus metadata in a vector database, with stable ids so updates replace instead of duplicate.

read path · on every question

  1. Embed the questionSame embedding model, so question and chunks share one space.
  2. Retrieve & filterTop-k nearest chunks, filtered by metadata, deduped, score-sorted.
  3. Augment the promptPaste the chunks in, labeled, with rules about staying inside them.
  4. GenerateThe model answers from the provided text — and says so when the answer is not there.
The write path runs when knowledge changes. The read path runs on every question. Most RAG quality problems live in the write path.

How you should implement it

Step 1: Decide what the knowledge base actually is

This is the step tutorials skip and the one that determines everything. RAG answers are only ever as good as what you indexed, so ask: what content do we trust enough to let a model repeat verbatim to customers?

  • Curated entries — FAQs, policy snippets, help-center articles a person wrote and owns. Highest trust, lowest volume. If this covers your questions, you have the easy case: skip crawlers and chunking entirely. Each entry is already a retrievable unit.
  • Document piles — wikis, PDFs, specs. This is where chunking (splitting documents into overlapping passages), parsing, and re-indexing pipelines earn their complexity. Also where stale and contradictory content sneaks in, because nobody owns individual paragraphs of a wiki.
  • Transcripts and tickets — tempting ("train it on our support history!") and usually wrong for retrieval: they mix one customer's situation with general advice, and they contain PII. Extract approved answers out of transcripts into curated entries instead.

Start with the smallest trustworthy set. A knowledge base of 80 owned FAQ pairs outperforms 8,000 crawled paragraphs for support work, because precision and accountability beat recall.

Step 2: Build the write path with stable identity

Whatever the source, every indexed unit needs: a stable id (so updating an entry replaces its vector instead of adding a near-duplicate — hash the natural key), the text to embed (for Q&A content, embed question and answer together, so "money back" can land on an entry whose question says "refund"), and metadata (category, tags, timestamps) for filtered queries. Pin your embedding model and record its name and dimensions; changing it later means re-embedding the entire index, because vectors from different models are not comparable.

Step 3: Build the read path to fail soft

Embed the question, query with a metadata filter, take the top handful, dedupe, sort by score, and join into a labeled block in the system prompt — with the two rules that make RAG honest: answer only from this material, and prefer the most specific, live source when sources disagree.

Then decide the failure behavior up front: retrieval is allowed to fail. Index down, embedding call erroring — catch it, log it, alert, and answer with an empty knowledge block under the no-inventing rules. A degraded cautious answer beats an error page, and it certainly beats a timeout.

Step 4: Build the editing loop before launch

A RAG system without an editing loop is a stale-answer generator with extra steps. A person must be able to: see which entries produced a given answer (log retrieved ids and scores alongside every reply), edit an entry by id, and delete one by id — with the index updating immediately. Retrieval does not fact-check your library; it repeats it with confidence. The delete button is a safety control.

How I implemented it on a real project

The support bot on the events platform I work on (the full build is here) runs what might be the smallest honest RAG system in production use: the knowledge base is only FAQs. No crawler, no chunker, no overlap windows, no nightly re-index. Here is how each decision above played out.

The write path: one FAQ, one vector

Each FAQ becomes a single string — FAQ QUESTION: … plus FAQ ANSWER: … — embedded with OpenAI's text-embedding-3-small at 1,536 dimensions into Pinecone. The id is faq_ plus the first 16 hex characters of a SHA-256 of the trimmed, lowercased question. Same question → same id → updates replace. Metadata carries the raw question and answer, the content string the model will read, a data-category marker, tags, and timestamps. Keys live in Secret Manager; the browser never touches Pinecone or OpenAI.

The editing interface is Slack, in the same channel where conversations are mirrored:

faq add
Can I get a refund?
Organizers set the policy on the event. If the event says no refunds, say that and point to Contact Host. Do not invent a deadline.

faq edit <id> and faq delete <id> complete the loop — and because every mirrored answer shows the retrieved ids and scores, "the bot said something wrong" turns into "delete faq_a1b2c3" in under a minute. That loop, not the vector math, is what makes the system trustworthy.

One FAQ, two journeys

a person writes

  1. faq add in the Slack thread: a question and its answer.
  2. The question is hashed into a stable id — same question, same vector.
  3. Question + answer are embedded together and upserted. Edits replace; deletes remove that one id.

a question arrives

  1. The question is embedded and the index is queried with a filter: FAQ data only.
  2. Matches are deduped, sorted by score, and the top few are pasted into the system message.
  3. If retrieval throws, the chat continues without the FAQ block — a logged alert, not an error page.

the vector index

Short FAQ pairs only — no document crawler, no overlap windows, no nightly reindex. A person owns every sentence in it.

Writes and reads meet in the vector index. The id is a hash of the question, so re-adding the same question updates one vector instead of duplicating it.

The read path: filtered, biased, and allowed to fail

General questions query with the raw question text, filtered to FAQ entries, deduped and score-sorted, top matches pasted into the system prompt as "matched FAQ entries, most relevant first" — with instructions to reuse FAQ wording, combine only details that agree, and prefer the live event page when it disagrees with an FAQ. The refund specialist prepends bias terms (refund, cancellation, policy, ticket dispute) so its neighbors are policy entries rather than random how-tos. Here is meaning-based matching on real examples:

Found by meaning, not keywords — try it

The visitor asks…

  • Refundsclosest match
  • How to publish
  • What each role can do
“Money back” and “Refunds” share no words. The question becomes a vector, and the index returns the FAQs that sit closest in meaning.

And the failure path is exactly as designed: a Pinecone error is caught and logged, an alert fires, and the chat continues with no FAQ block under the no-inventing rules.

What this deliberately is not

It is not an enterprise knowledge base over everything the company ever wrote — that would trade accountability for coverage. It is also not a substitute for records: "which ticket do I have?" is answered by a database read at request time, never by retrieval, because facts about one user are the wrong shape for a document index. FAQs explain the product; records explain the order.

Pitfalls to avoid

  • Indexing content nobody owns. Every entry needs a person who can be asked "is this still true?"
  • Chunking curated content. If entries are already short, chunking only adds ways to retrieve half an answer.
  • Random ids. Without stable ids, every edit adds a near-duplicate, and old answers keep winning retrieval.
  • Treating retrieval as infallible. Log the misses. If the right entry exists but scores low, rewrite its question text — that is tuning, and it is free.
  • Making retrieval load-bearing for uptime. The bot must answer, carefully, with the index down.

FAQ

At small scale, no — pgvector inside the database you already run is fine, and a few thousand vectors can even live in memory. Managed indexes earn their keep with scale and filtering features, not at day one.

  • #RAG
  • #Embeddings
  • #VectorDatabase
  • #Guide
  • #CaseStudy
Rushit Jivani headshot

Written by

Rushit Jivani

Senior Software Architect & AI Application Engineer. Architecture that shows up in crash rate, performance and how fast the team ships.