Key takeaways
- RAG (retrieval-augmented generation) retrieves relevant passages from your documents and gives them to a language model so answers are grounded in your data.
- It reduces made-up answers, keeps information current without retraining and allows citations.
- RAG is usually better than fine-tuning for company knowledge; fine-tuning suits style and specialised behaviour.
- Quality depends on good chunking, retrieval, access control and continuous evaluation.
- Start with one well-scoped knowledge source and a test set of real questions.
The problem RAG solves
Large language models are trained on public data up to a cutoff date. They do not know your pricing, policies, product documentation or customer records, and when asked about them they may produce confident but wrong answers. Retraining a model on your data is expensive and goes stale quickly. Retrieval-augmented generation, or RAG, solves this by looking up the relevant information at the moment a question is asked and giving it to the model as context.
How RAG works, step by step
- Ingest. Collect your sources: help-center articles, PDFs, wikis, tickets, product data.
- Chunk. Split documents into passages of a few hundred words that each make sense on their own.
- Embed. Convert each chunk into a numeric vector that captures its meaning.
- Store. Save the vectors in a vector database or a search index, along with metadata such as source, date and access permissions.
- Retrieve. When a user asks a question, convert it to a vector and find the most relevant chunks, often combining semantic and keyword search.
- Generate. Send the question and the retrieved passages to the language model with instructions to answer only from that material and cite sources.
The result is an answer grounded in your documents, with links back to where it came from.
A practical example
An employee asks, "How many vacation days do contractors in Germany get?" A plain model would guess. A RAG assistant retrieves the German contractor policy and the leave FAQ, drafts an answer from those passages and cites both documents. When HR updates the policy, the assistant reflects the change as soon as the document is re-indexed, with no retraining.
RAG vs. fine-tuning
| RAG | Fine-tuning | |
|---|---|---|
| What it changes | What the model can look up | How the model behaves |
| Best for | Company knowledge, documents, changing facts | Tone, format, specialised tasks |
| Keeping information current | Update the index | Retrain the model |
| Citations | Natural | Difficult |
| Upfront effort | Moderate | Higher |
| Access control per user | Possible with metadata filters | Not possible |
Where RAG projects go wrong
- Poor chunking. Passages that cut off mid-thought, or that are too long, retrieve badly. Structure-aware chunking helps.
- Weak retrieval. If the right passage is not retrieved, the model cannot use it. Hybrid search and re-ranking improve recall.
- Stale or duplicated content. Old documents produce old answers. Set refresh schedules and remove duplicates.
- Ignoring permissions. A RAG system must respect who is allowed to see which documents, or it can leak confidential information.
- No evaluation. Without a test set of real questions, you cannot tell whether a change made answers better or worse.
How to measure a RAG system
Track three things: whether retrieval finds the right passages, whether the answer is faithful to those passages, and whether users find it helpful. Build a set of 50 to 200 real questions with known good sources, run it after every change, and review failures by hand. Add user feedback buttons in production to catch problems the test set misses.
How to get started
Pick one knowledge source with clear ownership, such as your support documentation. Define twenty to fifty real questions, build a simple pipeline, and test with a small group of users. Once accuracy is reliable, expand to more sources and connect the assistant to actions, which is where RAG becomes part of a full AI agent.
Qodexify is a remote-first team of senior engineers building web apps, AI agents, workflow automation and websites. Learn more about our AI agent development service or talk to an engineer.
