Reranking in an n8n RAG pipeline: why the vector search top-k isn't enough

A RAG chatbot wired to Supabase pgvector sometimes answers off the mark even though the right information is genuinely in the document base — it just came back in position 8 or 12 of the match_documents call, never injected into the prompt because only the top 4 or 5 results were kept. The problem isn't the LLM, nor the document chunking: it's that vector search alone isn't built to finely rank results that are already close to each other. A reranking step between search and generation fixes exactly that weak point.

The tradeoff every vector search makes

An embedding encodes a piece of text into a single vector, computed once and for all, independently of whatever question will be asked of it later. That's what makes vector search fast: comparing a question against thousands of pre-computed vectors only costs one cosine similarity per candidate. But that speed comes at a cost — the model producing the embeddings, a bi-encoder, never sees the question and the passage together at encoding time. The paper that popularized this architecture, Sentence-BERT by Reimers and Gurevych (EMNLP 2019, see on Google Scholar), states it plainly: this is exactly what cuts the time to find the best match among thousands of texts from tens of hours down to a few seconds — at the cost of a less fine-grained ranking than a direct, pairwise question-to-passage comparison.

The practical consequence in an n8n RAG pipeline: the top 20 returned by match_documents almost always contains the right passage, but rarely in first position. If the prompt only keeps the raw top 4 or 5 results, part of the correct answers gets discarded before ever reaching the LLM.

Reranking: a second model, slower but more precise

A reranker is a cross-encoder: unlike a bi-encoder, it takes the question and each candidate passage together as input, and computes a relevance score specific to that pair. Nogueira and Cho showed as early as 2019 with their BERT reranker (“Passage Re-ranking with BERT” — Google Scholar) that this approach clearly improves ranking quality over vector similarity alone on passage-retrieval benchmarks like MS MARCO — at the cost of one computation per pair, making it impossible to apply across an entire document base, but a perfect fit for a small, already-preselected candidate set.

That's exactly the two-stage retrieval architecture to reproduce in n8n:

1. Wide vector search — retrieve 20 to 50 candidates with match_documents instead of the usual 4-5, to maximize the odds the right passage is in the set.
2. Reranking — a cross-encoder scores each candidate against the exact question asked.
3. Final selection — keep only the top 3-5 passages after reranking to build the prompt.

A concrete implementation in an n8n workflow

Building on the pipeline described in our Supabase pgvector RAG guide, reranking slots in between search and generation, without touching ingestion or chunking:

1. Supabase / Postgres node — call match_documents with match_count: 25 (instead of 5) to get a wide candidate shortlist.
2. HTTP Request node — send the question and the 25 passages to a reranking API. With the Cohere Rerank API, for instance, the request body is simply {"query": "...", "documents": ["chunk 1", "chunk 2", ...], "top_n": 5}, which returns the indices sorted by relevance score directly.
3. Code node — reconstruct the original chunks with their metadata (source, page) from the returned indices, so the citation information used in the chatbot with citations isn't lost.
4. AI node (LLM chain) — inject only these 5 re-ranked passages into the final prompt, as in a standard RAG pipeline.

For a self-hosted deployment with no third-party API dependency, a lightweight open-source cross-encoder model (ms-marco-MiniLM-L-6-v2, for example) can run behind a small Python API called by the same HTTP Request node — the rest of the n8n workflow doesn't change.

Passage order in the prompt matters too

Reranking passages isn't just about which ones to keep — the order they're then injected into the prompt in also affects answer quality. An LLM makes better use of information placed at the beginning or end of the context than information buried in the middle — a phenomenon documented in Liu et al.'s "Lost in the Middle" (TACL, see on Google Scholar). Concretely, in the Code node that builds the prompt: place the passage the reranker scored highest first (and optionally restate its key point in the final instruction) rather than leaving the raw order from the vector store, which has no reason to match actual relevance.

Cost and latency: when it's worth it

Reranking 20 to 50 candidates typically adds 200-500 ms per request — negligible next to the several seconds the LLM already takes to generate a response, provided you follow good rate-limiting practices if request volume is high. On cost, the Cohere Rerank API bills per document evaluated rather than per token, which stays marginal compared to the cost of generating a full LLM response.

Reranking isn't always necessary, though. On a small, homogeneous document base (a few dozen well-chunked pages), vector search alone already surfaces the right passages near the top most of the time — adding another stage just adds complexity for a marginal gain. Reranking starts paying off once the document base grows, diversifies (multiple document types, multiple sources), or user feedback flags off-target answers even though the information genuinely exists somewhere in the base.

Common pitfalls

• Reranking too few candidates: going from 5 to only 8-10 candidates fed into the reranker changes almost nothing — the benefit comes from starting with a wide set (20-50) to actually give the right passage a chance of being present.
• Losing metadata along the way: the reranking API returns scores tied to indices or raw text; if metadata (source, page) isn't correctly reattached after reranking, the citations shown to the user become wrong or disappear entirely.
• Reranking the whole document base: running a cross-encoder directly against thousands of documents instead of an already-preselected candidate set destroys latency — reranking only makes sense as a second stage, never as a replacement for vector search.
• Ignoring the score threshold: a reranker assigns a score to every candidate, including bad ones; without a minimum threshold, a clearly off-topic passage can still get injected into the prompt if there aren't enough good candidates available.

Going further

The RAG Assistant Pack (€119) provides the four workflows for ingestion, chat with citations, Notion sync, and a question-answering API already described in our guide on pgvector and choosing between HNSW/IVFFlat indexes — a reranking stage like this one plugs in directly downstream of the provided vector search node, without changing the rest of the pipeline. If your document assistant sometimes answers off the mark despite a well-built document base, this is often the first thing to try before revisiting chunking or switching embedding models.