Detecting fuzzy duplicates in n8n with embeddings

n8n's Remove Duplicates node, covered in our dedicated guide, handles the vast majority of strict duplicates — but it has a structural limit: its comparison is exact. Two records — "Jean Dupont, jean.dupont@acme.fr" and "J. Dupont, jdupont@acme.fr" — almost certainly refer to the same person, yet no field matches. These are fuzzy duplicates (or near-duplicates), and they multiply wherever data comes from multiple sources: web forms, CSV imports, synced CRMs, customer reviews typed in by hand. This guide shows how to catch them in n8n with embeddings, the same technology already used on this blog for document RAG and semantic caching.

Where exact matching fails

Exact matching — even normalized to lowercase and trimmed of extra whitespace — stays blind to three very common kinds of variation:

• Typos and data-entry mistakes: "Martn" for "Martin," an email misread off a business card;
• Legitimate spelling variants: "Société Générale" vs. "SG" vs. "Societe Generale SA," the same contact with or without a title;
• Rephrasings: two support tickets describing the same bug in different words, two customer reviews expressing the same complaint without sharing a single identical field.

In all three cases, what needs comparing isn't a string of characters but a meaning — exactly the problem embeddings already solve for document search.

The principle: embeddings and cosine similarity

The principle mirrors the semantic cache already described on this blog, applied to records rather than questions:

1. A record's discriminating fields (name, email, company, or a ticket's text) are concatenated into a short string.
2. That string is turned into an embedding via a model like text-embedding-3-small — see our guide to choosing an embeddings model for weighing cost, quality, and hosting.
3. The resulting vector is compared by cosine similarity against those already indexed for existing records.
4. Above a threshold, the new record is treated as a probable duplicate; below it, it's inserted normally and its vector is added to the index for future comparisons.

This approach rests on well-established NLP research. Nils Reimers and Iryna Gurevych's paper, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (EMNLP, 2019), shows that cosine similarity between sentence embeddings produced by a model trained for this purpose closely tracks human judgments of semantic closeness — exactly what's being exploited here to link two records phrased differently. On the more specific ground of record linkage (entity matching), Sidharth Mudgal and coauthors, in Deep Learning for Entity Matching: A Design Space Exploration (SIGMOD, 2018), systematically compare embedding-based architectures against classic deduplication methods and confirm their clear edge on noisy textual data — the typical case for a CRM fed from several sources.

Building the pipeline in n8n

The workflow sits ahead of the database write, on the ingestion path for a lead, a contact, or a ticket:

1. Trigger — a form webhook, a Google Sheets/Airtable polling node, or an incoming email.
2. Edit Fields node — basic normalization (lowercase, trimming extra whitespace) and concatenation of the key fields into a single string (${name} ${email} ${company}).
3. Embeddings node — vectorizing that string.
4. Vector search (KNN) — querying the existing index (pgvector or Qdrant, see below) to retrieve the closest candidate(s) and their score.
5. If / Switch node with three branches based on the score:
   • Above the high threshold → near-certain duplicate: the record is merged or auto-rejected, as documented in our article on syncing CRMs;
   • Gray zone → ambiguous score: sent for human approval via Slack before a decision;
   • Below the low threshold → new record: inserted into the database, with its embedding indexed for future comparisons.

This vector filtering step runs downstream of a cheap, exact first pass: strict normalization of the email or domain, comparable to the Remove Duplicates node already in place. Reserving vector search — heavier on compute and API calls — for the records that survive this first filter keeps the pipeline fast at volume while still catching the subtle cases.

Where to store and query the vectors

Two options cover most cases already documented on this blog. For infrastructure already built around Supabase, pgvector with an HNSW index answers in a few milliseconds even across several hundred thousand records, and sits naturally alongside your other tables (contacts, tickets, orders). For higher volume or a workload isolated from the rest of the database, Qdrant offers richer metadata filtering — useful for comparing, say, only the records belonging to a given client in a multi-tenant setup. Either way, the source record's id should be stored alongside the vector, so a matched candidate can be traced straight back to the full record.

Calibrating the threshold: trickier than for a semantic cache

On a semantic Q&A cache, a too-permissive threshold at worst answers off-target. Here, a too-permissive threshold merges two distinct people or companies — a far costlier mistake to unwind afterward in a CRM. Conversely, a too-strict threshold lets through the real duplicates you were trying to catch in the first place.

A reasonable starting point sits between 0.88 and 0.93, noticeably lower than the 0.95-0.97 used for a Q&A cache — because the goal here is precisely to catch the spelling variants that a semantic cache is trying to avoid conflating. Build a small sample of known real duplicates and misleading near-duplicates (two different employees at the same company, for instance) and tune the threshold against it before any rollout, following the same testing discipline described in our article on n8n evaluations for AI workflows.

A concrete case: deduplicating inbound leads

An SMB receiving leads from a web form, a trade show (CSV import), and a business partner (API) quickly accumulates records that refer to the same person in different forms. The pipeline described here slots naturally into a flow of automatic lead enrichment or AI-based qualification: before enriching and scoring a lead, you check whether it's already known under another form. The payoff is twofold — a sales rep doesn't follow up twice with the same person, and qualification metrics aren't inflated by duplicates padding out the volume.

Common pitfalls

• Concatenating too much noise into the embedded text: including a technical ID or a timestamp in the vectorized string dilutes the useful signal and skews similarity — keep only the fields that are genuinely discriminating.
• Comparing a single candidate: searching only for the nearest neighbor (top-1) hides cases where two or three existing records are all plausible matches; retrieving the top 3 to 5 candidates gives far more reliable context for the decision, whether human or automated.
• Ignoring an embeddings model change: as with any RAG pipeline, switching models invalidates the already-indexed vectors — the entire history needs re-embedding, not just new records.
• Merging without a trail: keeping the merged record's identifier (and the score that triggered the merge) makes it possible to roll back a mistake — a silent, irreversible merge is the worst-case scenario for a false positive.

Going further

The vectorization infrastructure described here — embeddings model, pgvector index, similarity search — is the same one set up in the RAG Assistant Pack (€119) for document ingestion: once that foundation is in place for one use case, it reuses directly for deduplicating leads, contacts, or tickets, with no new infrastructure to operate. If your immediate priority is instead sorting and prioritizing incoming email, the Inbox AI Pack (€79) covers that first link before duplicates even become a question.