# How RAG Works and Where It Still Breaks

You ask an AI assistant about something that happened yesterday, and it answers like it's still stuck six months in the past, because it is. A large language model doesn't know things the way we do. It's more like a student sitting an exam with the textbook closed, recalling what it memorized months ago, and sometimes getting it confidently wrong.

This isn't a one-off glitch. Every large language model is trained on a fixed snapshot of data, and then it's frozen. It doesn't check the news, browse the web, or update itself. It just predicts, based on whatever it absorbed before its training ended. Ask people in AI and they'll call this a "**knowledge cutoff" ,** the point where the model's information just stops. Ask it about the latest version of a library you're using, and it might confidently describe features from two releases ago, or worse, mix up syntax from a version that no longer exists. So what if, instead of relying purely on memory, the model could look something up right before answering?

## What RAG Is, and Why It Was Introduced

This is the exact problem RAG was built to solve. **RAG** stands for **Retrieval-Augmented Generation**, but forget the acronym for a second, the idea is simple. Instead of expecting the model to remember everything, give it the right information right when it needs it. Same closed book student from before, except now, seconds before the exam starts, someone slides the right page of the textbook across the desk. The student still has to read and reason well, but at least they're not guessing from memory alone.

## How a Basic RAG Pipeline Works

A RAG system actually works in two separate stages, and it helps to think of them as **"getting ready"** and **"answering."**

The getting ready stage happens before anyone even asks a question. All your documents, whether that's a PDF, a website, or internal notes, get broken down into smaller pieces called **chunks**, since feeding a whole 50-page document to a model at once isn't practical. Each chunk is then converted into a set of numbers called an **embedding**, basically a fingerprint that captures what that chunk *means*, not just the words in it. These fingerprints get stored in a special kind of database called a **vector database**, built specifically to store and search these number fingerprints quickly. That's the setup.

Now, the answering stage kicks in the moment a user asks something. Their question also gets turned into the same kind of fingerprint, and the system searches the vector database for chunks whose fingerprints look the most similar, this step is called a **similarity search**. Whatever chunks come back closest in meaning get pulled out and handed to the model, along with the original question. Only then does the model generate its answer, using the retrieved chunks as backup instead of relying purely on what it memorized during training.

![](https://cdn.hashnode.com/uploads/covers/6a4330461986d789d834671e/1a3f0c4a-45fc-4702-92b8-73ce41450c0a.png align="center")

***User Query → Retrieval → LLM → Response***

## Common scenarios where RAG works well

RAG tends to shine anywhere a business has a pile of specific, private information that a general AI model was never trained on. A support chatbot that answers questions using a company's own help docs is a classic case, it can quote the actual refund policy instead of guessing. "Chat with your PDF" tools work the same way, upload a research paper or a contract, and the model answers questions using that exact document instead of general knowledge. Internal company search is another common one, employees asking things like "what's our leave policy" get answers pulled straight from HR documents, not from whatever the model picked up during training.

But having the right information available doesn't always mean the model uses it correctly, and that's where things start to get interesting.

## Why RAG Sometimes Gives Incorrect Answers

Adding retrieval to a model fixes one problem, but it quietly creates a new expectation, that the answer will now be correct. It won't always be. RAG only guarantees that the model *saw* relevant information before answering, not that it understood it correctly, used all of it, or that the information itself was accurate and complete in the first place. Think of it like handing someone the right page of a textbook right before an exam, they can still misread a line, skip a paragraph, or mix it up with something they already believed. The rest of this depends on things most people never think about: whether the system actually found the right information, whether that information was chunked in a way that kept its meaning intact, whether it all fit into what the model can process at once, and whether the source document itself was even up to date.