The complaint usually arrives as “the model is hallucinating”. Nine times out of ten it is not. The model is answering faithfully from the passages it was given, and those passages were the wrong ones.
That distinction matters because it changes where you look. Prompt engineering will not rescue bad retrieval. Below are the questions worth asking, in the order that finds the problem fastest.
First: is the correct passage in the index at all?
Before debugging ranking, confirm there is something to rank. Take a query that fails, find the document that should answer it, and check whether its text actually made it into your vector store.
This fails more often than people expect, and rarely because of the embedding model. PDF extraction silently drops text in multi-column layouts. Tables become word soup. Scanned documents produce nothing at all without OCR. Ingestion jobs partially fail and nobody checks the count.
# Search your store for the literal text, no embeddings involved
hits = store.scroll(filter={"text": {"$contains": "renewal notice period"}})
print(len(hits)) # 0 means this is an ingestion bug, not a retrieval bugIf the text is not there, stop. No amount of reranking or query rewriting will retrieve a passage that does not exist, and this is the single most common root cause of “our RAG does not work”.
Second: is it in the index but ranking badly?
If the text exists, retrieve a large number of results and find its actual rank. If the right chunk is at position 40 and you retrieve the top 5, you have a ranking problem — a genuinely different problem from the one above, with different fixes.
results = store.similarity_search(query, k=100)
for i, r in enumerate(results):
if "renewal notice period" in r.page_content:
print(f"correct chunk ranked #{i}") # #40 = ranking, not ingestionWhere it lands tells you what to do. Near the top but outside your k? Raise k and add a reranker. Buried in the tail? Your embeddings are not capturing the relationship at all, and you need to look at chunking or the model itself.
Third: are your chunks split in a way that destroys meaning?
Fixed-size splitting is the default in every tutorial and it is frequently the culprit. Cutting at 512 characters regardless of structure severs sentences mid-clause and separates a heading from the content beneath it.
The failure mode is specific: a chunk that says “This must be completed within 30 days” is useless when the sentence naming what must be completed sits in the previous chunk. It will not match the query, and if it does, it will not answer it.
Split on structure where you have it — headings, sections, paragraphs — rather than on character counts. Add a modest overlap so a sentence spanning a boundary appears in both. And consider prepending the document title and section heading to each chunk’s embedded text, so a chunk carries the context of where it came from. Our advanced chunking strategies guide goes deeper on the trade-offs between strategies.
Fourth: is it a vocabulary mismatch rather than a semantic one?
Dense embeddings are good at meaning and surprisingly weak at exact tokens. Part numbers, error codes, internal acronyms, surnames — these often retrieve badly because the embedding space does not treat “ERR_4417” as meaningfully different from “ERR_4418”.
If your failing queries contain identifiers, that is the diagnosis. The fix is hybrid search: run BM25 keyword matching alongside vector search and fuse the results.
from rank_bm25 import BM25Okapi
def hybrid(query, k=10):
dense = store.similarity_search(query, k=50)
sparse = bm25.get_top_n(query.split(), corpus, n=50)
# Reciprocal rank fusion — no score normalisation needed
scores = {}
for rank, d in enumerate(dense):
scores[d.id] = scores.get(d.id, 0) + 1 / (60 + rank)
for rank, d in enumerate(sparse):
scores[d.id] = scores.get(d.id, 0) + 1 / (60 + rank)
return sorted(scores, key=scores.get, reverse=True)[:k]Reciprocal rank fusion is the pragmatic choice here because it combines rankings without needing the two systems’ scores to be on comparable scales, which they are not.
Fifth: are you embedding the query the same way you embedded the documents?
A quiet, embarrassing one. Several embedding models are asymmetric and expect a prefix distinguishing queries from passages — omit it and every similarity score is subtly wrong. Check your model’s documentation for whether it wants query: and passage: prefixes.
Also verify you are using the identical model version for indexing and querying. Re-embedding your corpus with a new model while the query path still uses the old one produces retrieval that is not broken so much as meaningless, and it produces no error at all.
Sixth: would a reranker fix it more cheaply than anything else?
If the correct chunk is reliably landing somewhere in your top 50 but not your top 5, add a cross-encoder reranker before you touch anything else. It is usually the highest-value change available and it requires no re-indexing.
The reason it works is architectural. Bi-encoders — ordinary embedding models — encode the query and the document separately, so they never compare the two directly and lose fine-grained relationships. A cross-encoder feeds both into the model together, which is far more accurate and far too slow to run across a whole corpus. Retrieving 50 candidates cheaply and reranking those with a cross-encoder gets you most of the accuracy at a fraction of the cost.
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
candidates = store.similarity_search(query, k=50) # cheap, recall-oriented
scores = reranker.predict([(query, c.page_content) for c in candidates])
top = [c for _, c in sorted(zip(scores, candidates), key=lambda x: -x[0])[:5]]The trade is latency: reranking 50 candidates adds real milliseconds. Measure whether that matters for your interface before assuming it does — for a chat UI where the model then generates for several seconds, an extra 80ms is invisible.
Seventh: are you filtering when you should be?
A surprising amount of “irrelevant results” is actually correct retrieval across the wrong scope. If your index spans multiple products, tenants, or document versions, a semantically perfect match from the wrong partition is still wrong.
Metadata filtering applied before the vector search solves this and costs almost nothing. Filter to the current version, the user’s tenant, the relevant document type — then search within that. Retrieving superseded policy documents alongside current ones is a common and entirely avoidable failure, and in a multi-tenant deployment it is a data-isolation problem rather than a relevance one.
Eighth: have you actually measured, or are you spot-checking?
This is the one that separates teams who improve their pipeline from teams who keep tinkering. Build a small evaluation set — thirty to fifty real questions with the passage that should answer each — and measure recall@k before and after every change.
Without it you are guessing. Chunk size, k, reranking, and hybrid weighting all interact, and a change that fixes the query you happen to be looking at frequently degrades others. Thirty labelled examples is enough to catch that, and it is an afternoon of work. Our RAG evaluation frameworks guide covers the tooling once you outgrow a spreadsheet.
The order that matters
Check ingestion before ranking, ranking before chunking, and measure before optimising. Most “RAG is not working” reports resolve at step one — the text was never in the index — and the teams that spend a week tuning embeddings before checking that are more common than anyone would like to admit.