Removing duplicates in n8n: the Remove Duplicates node and its cross-execution memory

Duplicates are the chronic disease of workflows: an RSS feed returning the same articles on every read, a replayed webhook, an export overlapping the previous one, two sources naming the same contact. And every uncaught duplicate costs something — an AI call paid twice, an email sent twice, a duplicated row in the database. n8n devotes a whole node to the problem, Remove Duplicates, whose most valuable feature is also its least known: persistent memory across executions. This guide covers its three operations, its traps, and the cases where a different strategy beats it.

Operation 1 — Deduplicating within the current execution

"Remove Items Repeated Within Current Input" compares the flow's items against each other and keeps one copy of each. Three comparison modes:

• All Fields: two items identical field for field are duplicates;
• All Fields Except: ignore noisy fields (collection timestamp, technical ID) that would make every item spuriously unique;
• Selected Fields: compare only the discriminating fields — email for contacts, url for articles.

It's the tool for the "two merged sources" case: after a Merge node combining lists from Airtable and a CSV, a Remove Duplicates on email cleans up the union. Classic trap: the comparison is exact. "Smith" and "smith " (capitalized, trailing space) are two distinct values — normalize upstream in an Edit Fields node (.toLowerCase().trim()) before comparing.

Operation 2 — Memory across executions: "Remove Items Processed in Previous Executions"

The feature that changes everything for scheduled workflows: n8n stores already-seen values in its database, from one execution to the next. You designate a key field, and three modes are available:

• Value Is New: passes only if the key has never been seen — the universal mode (article URL, order ID, email);
• Value Is Higher than Any Previous Value: for growing cursors — auto-incremented IDs, invoice numbers — n8n only keeps the maximum seen, leaner than a list;
• Value Is a Date Later than Any Previous Date: same logic on a timestamp (updated_at).

This is the node that makes RSS monitoring trivial ("which articles are new since yesterday?"), along with polling an API that lacks a reliable since parameter, or incremental table syncs. The History Size option caps how many values are kept in "Value Is New" mode: size it beyond the number of items a feed can plausibly return between two passes, otherwise the oldest values fall out of memory and reappear as "new".

Two properties worth memorizing: the history is per node (two Remove Duplicates nodes share nothing, even configured identically), and it survives restarts since it lives in n8n's database — but not a key-field change, which starts from scratch. After testing, remember the same node's "Clear Deduplication History" operation: without that cleanup, values seen during tests silently punch holes in production.

What Remove Duplicates doesn't solve

The node deduplicates what flows through it. Three blind spots, each with its remedy:

• The replayed webhook: two deliveries of the same event arrive in two simultaneous executions — the second can pass before the first has written its history. For webhooks, the real answer is an idempotency key checked in a database, atomic by construction;
• The duplicate already in the database: Remove Duplicates only knows its own history, not the state of your CRM. The final guarantee is a unique-key upsert on the destination side — INSERT ... ON CONFLICT in Postgres, upsert by External ID in Salesforce, search-before-write in HubSpot/Pipedrive;
• The fuzzy duplicate: "John Smith / john.smith@" and "J. Smith / jsmith@" are the same person, but no field equality will say so. That's record linkage territory — aggressive normalization, fuzzy comparison, even LLM judgment on candidate pairs. The topic is a research field of its own: the reference survey by Ahmed Elmagarmid, Panagiotis Ipeirotis and Vassilios Verykios, "Duplicate Record Detection: A Survey" (IEEE Transactions on Knowledge and Data Engineering, 2007, see on Google Scholar), maps these techniques for detecting non-identical duplicates — and usefully reminds us that no exact method catches them all.

In practice, the healthy architecture layers the levels: dedupe early in n8n to avoid reprocessing and re-paying (every duplicate discarded before an AI call is a call saved), lock down in the database for final integrity.

The alternatives, by context

• Code node: for genuinely specific duplicate logic (sliding windows, "business-sense" duplicates), a JavaScript Set over a computed key does the job in ten lines — see expressions and the Code node;
• SQL at the source: if the data comes from a database, a SELECT DISTINCT ON (...) or GROUP BY deduplicates before it even enters n8n — always cheaper than transporting duplicates;
• Dataset comparison: the Compare Datasets node identifies additions, deletions and changes between two lists — for when the question is no longer "which duplicates?" but "what changed?".

In short

Remove Duplicates has two faces: cleaner of a single execution (comparison on chosen fields, after normalization) and memory across executions ("Value Is New" on a stable key, a well-sized History Size, a purgeable history). It saves reprocessing and paid calls; it replaces neither the idempotency key for concurrent webhooks, nor the database upsert for final integrity, nor record linkage for fuzzy duplicates. Layer all three and duplicates stop being a fate and become a handled case.