Evaluating RAG quality in n8n: metrics, test sets and the LLM-as-a-judge pattern

A RAG built in n8n almost always makes a great first impression: three questions, three correct answers, ship it. Three months later, users complain about off-target answers — and nobody can say when quality dropped, because nothing ever "broke" in the technical sense. A RAG doesn't crash: it degrades silently, as documents pile up, prompts get tweaked and models get updated. The only defense is to measure: simple metrics, a reference test set, and an evaluation workflow replayed after each change. Here is how to set that up in n8n.

Why a RAG degrades without warning

A RAG pipeline stacks stages that each drift independently: the document base grows and dilutes the good passages, a badly chunked batch of new documents pollutes retrieval, a "harmless" prompt tweak changes the model's behavior, the provider updates its LLM without notice. None of these events triggers an error. And manually testing three questions catches nothing: the ones you know by heart keep working — it's the other forty that degrade.

The difficulty specific to RAG: when an answer is bad, there are two possible culprits. Either the search failed to surface the right passages (a retrieval problem), or it surfaced them but the model turned them into a poor answer (a generation problem). Without separate metrics for these two stages, you fix things at random — rewriting the prompt when the real problem lies in the chunking, or the other way around.

The metrics that matter, explained simply

Retrieval side: are the right passages coming back?

• Hit rate — does the passage containing the answer appear in the returned top-k? The simplest and most telling metric: a 70% hit rate means 3 questions out of 10 are lost before they even reach the LLM.
• Context recall — what share of the information needed to answer is present in the retrieved passages? Useful when a complete answer spans several chunks.
• Context precision — of the retrieved passages, how many are actually useful? Low precision means a prompt stuffed with noise that distracts the model.
• MRR (Mean Reciprocal Rank) — at what average position does the first relevant passage appear? A good passage ranked 8th counts toward the hit rate but reveals mediocre ranking.

Generation side: is the answer any good?

• Faithfulness to the context — does the answer rely solely on the provided passages, or did the model add claims of its own? The anti-hallucination metric par excellence.
• Answer relevance — does it actually address the question asked, without digressions?
• Correctness — compared against a human-written reference answer, is it factually right?

This split is not a homemade invention: it is the structure proposed by the RAGAS framework, described in the study "RAGAS: Automated Evaluation of Retrieval Augmented Generation" by Es, James, Espinosa-Anke and Schockaert, published in 2023 (see it on Google Scholar), which formalizes faithfulness, relevance and context quality as automatically evaluable dimensions. RAGAS is the field's reference framework if you want to dig deeper; for an n8n RAG, the same principles apply without installing anything.

Building the reference test set

The whole setup rests on a set of 20 to 50 question-answer pairs stored in a Google Sheet or an n8n Data Table: the question, the expected answer (2-4 sentences written by someone who knows the subject), and ideally the identifier of the source document — that's what makes the hit rate computable.

Three rules to make this test set worth anything:

1. Start from real questions. Mine the chatbot logs, support tickets, incoming emails. Invented questions are too clean; real ones contain typos, vague phrasing and exact references.
2. Include hard cases. Answers spread across several documents, precise product codes, and above all 3 to 5 questions with no answer in the knowledge base — the expected answer is then "I don't know", and any RAG that invents something instead must be penalized.
3. Grow it with every incident. Every bad answer reported in production becomes a new row in the test set: that's what keeps the same bug from coming back.

The LLM-as-a-judge workflow in n8n

The principle: an evaluation workflow reads each row of the test set, calls the real RAG pipeline (isolated in a sub-workflow so the logic isn't duplicated), then hands everything to a judge LLM that scores the answer against an explicit rubric.

The judge's prompt is the critical part. A vague instruction ("rate this answer") produces irreproducible scores; a detailed rubric with structured output, made reliable with the Structured Output Parser, yields scores you can actually use:

You are a strict evaluator. You are given a QUESTION, the retrieved
CONTEXT, the generated ANSWER and a REFERENCE ANSWER.

Score each criterion from 1 to 5:
- faithfulness: is every claim in the answer supported by the
  context? (5 = fully grounded, 1 = obvious fabrications)
- relevance: does the answer address the question asked?
- correctness: is the answer consistent with the reference?

Reply ONLY in JSON:
{"faithfulness": n, "relevance": n, "correctness": n, "verdict": "pass|fail", "comment": "one sentence"}

The approach is empirically validated: the study "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" by Zheng et al., published at NeurIPS 2023 (see it on Google Scholar), shows that a strong LLM judge reaches over 80% agreement with human evaluators — the level of agreement humans reach with each other — while documenting its biases (position, verbosity, self-enhancement). Hence two precautions: use a judge from a different family than the generator model, and manually spot-check a sample of its scores from time to time.

Rather than wiring everything by hand, lean on n8n's native Evaluations: the Evaluation Trigger node reads the dataset, the Evaluation node records the metrics, and the Evaluations tab keeps a history of every run's scores — our guide to n8n Evaluations covers this setup, where the judge rubric above plugs in as a custom metric. Also think about tracking the cost of the AI calls these campaigns make: modest over 50 questions, but worth knowing.

What to fix, depending on the diagnosis

This is where the retrieval/generation split pays off, because it dictates which project to open:

• Low hit rate or recall → the problem sits upstream of the LLM. Revisit the document chunking, check the embedding model, and if failures involve exact references (codes, SKUs), move to hybrid search. Mixed sources? Metadata filtering narrows the search space.
• Decent hit rate but low MRR or precision → the right passages come out, but poorly ranked or drowned in noise: the textbook use case for reranking.
• Good retrieval, low faithfulness or correctness → the problem is in generation: a prompt that's too permissive (add a strict instruction — "answer only from the context; otherwise say you don't know" — as in our Supabase RAG guide), or a context so long it buries the information — inject fewer passages, not more.
• Low relevance across the board → read the judge's comments case by case before touching the pipeline.

Tracking over time: the before/after routine

A single evaluation is a snapshot; the value is in the film. The routine: one score before, one score after, for every change. Changing the chunking? Run before, run after, compare the averages. Hit rate jumps from 72% to 86% but faithfulness drops? You see it immediately, instead of discovering it in complaints three weeks later. Keep every run with its date and a description of the change, and schedule a periodic run even without changes: that's the one that catches silent drift, whether a provider-side model update or the evolution of your document base.

Key takeaways

• A RAG doesn't break, it degrades silently: without measurement, your users will be the ones to tell you.
• Separate retrieval (hit rate, recall, context precision, MRR) from generation (faithfulness, relevance, correctness): the diagnosis dictates the cure.
• A test set of 20 to 50 pairs drawn from real questions, hard cases and unanswerable questions included.
• The LLM-as-a-judge pattern with a structured rubric automates scoring, within n8n's native Evaluations.
• Weak retrieval → chunking, embeddings, hybrid search, reranking. Weak generation → prompt, shorter context.
• Measure before/after every change and on a schedule.