Three words. Three steps. Embed the query, find similar text, feed it to a model.
The acronym makes it sound like plumbing. And if you wire it up exactly that way, it will work on your first ten test questions. You'll demo it to someone and feel good about it.
Then a real user types "the bot doesn't respond after I set up the webhook" and your three-step pipeline pulls chunks about webhook configuration, chunks about bot response templates, and chunks about Slack integration timeouts. Every chunk scored high on vector similarity. None of them hold the answer. The answer lives in the connection between a webhook misconfiguration and a specific error handler that silently swallows messages.
Vector search found text that resembles the question. It missed the answer entirely.
This is where the tutorials end and the actual work starts.
The Problems Behind the Acronym
We've been running a RAG pipeline in production at Well Digit, and every non-obvious failure we've encountered falls into one of five buckets. All five are problems the basic three-step model pretends don't exist.
1. The classification problem.
Not every user message deserves a retrieval cycle. Someone sends "thanks, that helped" and the system spins up a full embedding, queries the vector store, retrieves documentation chunks about gratitude, and generates a paragraph about how glad it is to help. Wasted tokens, wasted latency, a strange moment for the user.
The first real decision in the pipeline should be binary: does this input require retrieval at all? Small talk skips everything and goes straight to generation. One gate. It saves compute on roughly 30% of messages in our case.
2. The ambiguity problem.
"How do I set up the integration?" Your product supports six integrations. The system embeds the query, pulls whichever integration's documentation happened to be closest in vector space, and responds with full confidence about the wrong one. The user has no way to know the answer is wrong because the system didn't pause to check.
That's a design failure, not a model failure. Before generation, you need to verify: do the retrieved documents converge on a single topic or scatter across several? If they scatter, the honest move is a clarifying question.
We handle this with graph cluster detection. If matched entities fall into disconnected clusters with no path connecting them within four hops, the query touches multiple unrelated topics. The system asks which one the user meant instead of guessing.
3. The context overflow problem.
You pull 20 relevant chunks. That's 15,000 tokens. You pack them all into the prompt. The model reads the beginning, reads the end, and drifts through the middle without registering it.
The "lost-in-the-middle" phenomenon is well-documented in research. Retrieving fewer chunks isn't the answer (you lose recall). Summarizing the retrieved context before generation is. Compress 15,000 tokens of raw material into 3,000 tokens of focused summary. The model actually processes the whole thing.
We cache these summaries by content hash. Same context set, same summary. Cuts redundant LLM calls on repeated questions without any extra logic.
4. The flat retrieval problem.
The big one.
Vector similarity operates on text surface. It locates chunks that sound like the query. But knowledge has structure. Causes lead to consequences. Prerequisites gate solutions. Alternatives branch from common roots.
"What happens when finding X is linked to corrective action Y?" — this is a relational question. No single chunk contains the answer because information about finding X sits in one document while corrective action Y lives in another. They're connected by a relationship, not by lexical proximity.
Vectors can't see relationships. Graphs can.
5. The intent problem.
A user asking "what is two-factor authentication?" and a user saying "2FA broke after the update" need fundamentally different retrieval strategies. The first is conceptual exploration. The second is a problem report. Pulling the same chunks for both yields mediocre results for both.
Classifying intent before retrieval lets you shape the search. A concept query retrieves explanatory content and related topics. A problem query retrieves symptoms, causes, and known resolutions. Same knowledge base. Different traversal logic.
What a Graph Adds to RAG
A knowledge graph stores entities and the relationships between them. Structure, not prose.
The simplest framing: vector search tells you "documents that discuss similar things." A graph tells you "things connected to what you asked about, and how they connect."
In practice, the graph sits alongside the vector store. Vector search finds entry points. The graph expands outward from those entry points, surfacing related knowledge that cosine similarity alone would never reach.
How it works in practice
Designing a Graph Schema for Your Domain
There is no universal graph schema. The entities you model and the relationships you define determine what the graph reveals that vectors miss. This is domain work, not framework work.
Two examples.
Customer support knowledge base
This is the domain we've built the most in.
Entities. The central node type carries a kind field that distinguishes: Problem, Solution, Concept, Feature, Configuration. A Problem is something that breaks. A Solution is how you fix it. A Concept is explanatory knowledge. Feature and Configuration describe the product structure.
Relationships. This is where the schema earns its value.
CAUSES — connects a root cause to a visible symptom. "Invalid webhook URL" CAUSES "Bot doesn't respond." When the user reports the symptom, the graph walks backward to the cause.
RESOLVED_BY — connects a problem to one or more solutions. "Bot doesn't respond" RESOLVED_BY "Verify the webhook endpoint is publicly accessible." Multiple solutions per problem. The model picks the most contextually relevant one.
DIAGNOSED_BY — connects a problem to a diagnostic question. "Bot doesn't respond" DIAGNOSED_BY "Does the webhook URL return 200 on a GET request?" This edge drives clarifying questions before the system attempts a full answer.
RELATED_TO — general bidirectional association. "Webhook configuration" RELATED_TO "API key setup." Used for concept-level exploration when the user is learning, not troubleshooting.
EVIDENCE_FOR and MENTIONS — the bridge between worlds. They connect chunks (the vector domain) to entities (the graph domain). A documentation chunk is EVIDENCE_FOR a Solution, or MENTIONS a Feature. Without these edges, the two retrieval systems never talk to each other.
Intent-aware traversal. The same graph responds differently depending on what the user needs. Problem intent follows CAUSES backward and RESOLVED_BY forward. HowTo intent follows RESOLVED_BY to solutions and looks for step-by-step attributes. Concept intent walks RELATED_TO broadly to map connected knowledge.
Same entities. Same edges. Different paths through them.
E-commerce product catalog
Different domain. Different schema. Same principle.
Entities. Product, Category, Feature, Specification, Review, Brand.
Relationships.
BELONGS_TO — Product to Category. "Running Shoe X" BELONGS_TO "Men's Athletic Footwear."
HAS_FEATURE — Product to Feature. "Running Shoe X" HAS_FEATURE "Carbon fiber plate." When someone searches "shoes with carbon plates," vector search catches the phrase. Graph traversal also finds every other product sharing that feature, including ones that describe it as "carbon-infused midsole" or "energy return plate." Different words, same feature node.
COMPATIBLE_WITH — Product to Product. "Running Shoe X" COMPATIBLE_WITH "Performance Insole Y." This knowledge doesn't exist in any single product description. It lives purely in the relationship.
COMPARED_TO — links competing products. When someone asks "which is better, X or Z?", the graph surfaces comparison context from both directions.
The pattern across both examples: entities represent the things users ask about. Relationships represent the connections between them that no individual chunk of text captures. The graph stores what falls between the documents.
When You Don't Need a Graph
Adding a graph increases complexity in indexing, storage, and query processing. It's worth it when your data has entities with meaningful relationships, when users ask relational questions, and when ambiguity is a frequent problem.
If your use case is searching 500 FAQ articles and returning the most relevant one, vector search handles it. Don't add a graph because it sounds architecturally interesting. Add it when your users ask questions that require traversing connections between things.
The Pipeline Is the Product
RAG is not three steps. The pipeline we run has sixteen stages. Every one of them exists because a concrete failure in production demanded it. Classification to avoid pointless retrieval. Intent detection to shape the strategy. Vector search for semantic proximity. Graph expansion for relational knowledge. Reranking to merge scoring signals. Summarization to respect attention limits. Ambiguity detection to avoid answering confidently and incorrectly. Model selection to balance quality against cost.
None of this was designed on a whiteboard. Each stage was added because something broke and we needed to stop it from breaking again.
The three-letter acronym sells simplicity. The work behind it is anything but.