RAG on your own website with n8n, Firecrawl and Supabase: a chatbot that answers from your pages

A small business with a decent website — service pages, product docs, FAQ, blog — has already written the answers to most of the questions its support team and sales reps receive. The problem isn't the content, it's access: nobody reads forty pages before writing an email. A RAG chatbot plugged into the site's content flips the burden: the visitor asks their question, the assistant retrieves the relevant passages from your pages and answers while citing them. First-line support, prequalifying inbound requests, internal help for new hires: same mechanics, same index.

We've already covered the building blocks one by one — connecting Firecrawl to n8n for scraping, RAG with Supabase pgvector for vector storage. This guide assembles them into one complete, specific use case: indexing your own site and keeping it up to date, which raises questions PDF ingestion never does — crawl scope, per-URL upserts, detecting unchanged content.

Why crawl your site rather than export the CMS

The first instinct is often to grab the content at the source: a WordPress export, a database query, the headless CMS API. It's almost always the wrong path, for three reasons:

• The published site is the source of truth. It's the content actually online, proofread, maintained — the one your visitors see. A CMS export can contain drafts, revisions, technical fields — and miss whatever a plugin or a page builder assembles at render time.
• Crawling is independent of the internal format. Whether the site runs on WordPress, Webflow, a static site generator, or a mix of all three, Firecrawl returns the same clean markdown for every page. Switch CMS next year and the ingestion pipeline doesn't change by a single line.
• The URL comes for free. Every crawled page arrives with its public URL — exactly the metadata the chatbot will need to cite its sources. A CMS export would force you to reconstruct it.

A direct export keeps one legitimate use case: unpublished content (internal procedures, a private knowledge base) that was never meant to be on the site. For everything that's online, crawl.

Step 1 — Crawl the site with Firecrawl

Firecrawl's Crawl operation walks the site by following links and returns every page as markdown stripped of boilerplate — menus, footers, cookie banners. The connection (native node on n8n Cloud, community node when self-hosted, or a direct HTTP Request with your credentials) is covered in our Firecrawl guide; here, let's focus on the settings specific to this use case.

Two settings matter more than all the others:

• Scope. The crawl stays within the starting domain by design, but tighten it further with include and exclude patterns: no point indexing the legal notices, the privacy policy, cart or account pages, or paginated blog archives. Every useless page indexed is noise in the search results and credits burned. A typical configuration looks like this:

{
  "url": "https://www.yoursite.com",
  "includePaths": ["/services/.*", "/docs/.*", "/blog/.*", "/faq.*"],
  "excludePaths": ["/legal.*", "/cart.*", "/my-account.*", "/blog/page/.*"],
  "scrapeOptions": { "formats": ["markdown"] }
}

• Asynchronicity. A crawl is an asynchronous job: you start it, then poll its status until completion before collecting the pages. In n8n, that's a Wait + status-check loop, or the node's Crawl Status operation.

On the way out, each item carries the page's markdown and its metadata — including the URL and title. Don't lose them: the entire rest of the pipeline depends on them.

Step 2 — Chunk by sections, not arbitrary blocks

Firecrawl's markdown has a decisive advantage over raw text: it's already structured by headings. Use that. Rather than blindly splitting every 1000 characters, split by sections — each ## or ### opens a chunk — and only re-split the sections that run too long. A visitor's question almost always targets a topic the writer already isolated under a heading: a chunk that follows the section boundaries has every chance of containing the full answer, where an arbitrary block cuts it in two. Our chunking guide compares the strategies in detail.

The non-negotiable part: every chunk must carry its metadata — the page URL, the page title, and the title of the section it came from. That's what will let the chatbot cite "the Pricing page, Commitment section" with a clickable link, instead of an unverifiable answer. In a Code node, the target structure per chunk:

{
  "chunk": "Text of the section...",
  "metadata": {
    "url": "https://www.yoursite.com/services/maintenance",
    "title": "Application maintenance",
    "section": "Response times"
  }
}

Step 3 — Embeddings and storage in Supabase pgvector

The embeddings + pgvector mechanics are those of our Supabase RAG guide; only the table changes slightly, promoting the URL to a proper column — you'll be filtering and deleting by URL on every update, so it might as well be an indexed column rather than a field buried in the JSON:

create extension if not exists vector;

create table documents (
  id bigserial primary key,
  url text not null,
  chunk text,
  embedding vector(1536),
  metadata jsonb
);

-- Index for per-URL deletes/upserts
create index idx_documents_url on documents (url);

-- Vector index (worthwhile beyond ~10,000 chunks)
create index on documents using hnsw (embedding vector_cosine_ops);

The ingestion workflow then chains: chunks → embeddings node (the same model as at query time, always) → insert into documents, with url pulled from the metadata and the rest kept in metadata. For a small-business site — a few dozen to a few hundred pages — the full ingestion takes minutes.

Step 4 — Keeping the index fresh: re-crawl, per-URL upsert, hashing

A site lives: pages get edited, posts get published, offers get pulled. A frozen index goes silently wrong — the chatbot will confidently cite an outdated price. The strategy that works, detailed in our updating a RAG index guide, comes down to three principles:

1. Scheduled re-crawl. A weekly Schedule Trigger reruns the same crawl. For most small-business sites, weekly is more than enough; go daily only if the content genuinely moves every day.
2. Upsert per URL, not per chunk. For each re-crawled URL: delete all its old chunks, then insert the new ones. Trying to update chunk by chunk is a trap — as soon as one paragraph moves, the entire page's splitting shifts. Delete-and-reinsert per URL is atomic at the scale that matters (the page) and trivial thanks to the url column:

delete from documents where url = $1;
-- then insert that URL's new chunks

3. Hashing to detect unchanged pages. Most pages don't change from one week to the next. Store a hash of each page's markdown (in a small pages_hash table or in the metadata), compare it to the hash from the new crawl, and only re-embed the pages whose hash changed. On a 200-page site where 5 pages change per week, that's the difference between 200 and 5 trips through the embeddings API — every week.

Step 5 — Querying: agent, citations, hybrid search

On the conversation side, the architecture is the one from the Supabase guide: a Chat Trigger, then an AI Agent equipped with the vector store as a retrieval tool (or a Question and Answer chain for a pure FAQ — simpler and more deterministic). What's specific to "website RAG" is the system message: the sources are public URLs, so demand them.

You answer only from the site content provided by the search tool.
End every answer with "Sources:" followed by the metadata.url values
of the passages used.
If the site doesn't cover the question, say so and suggest contacting
the team, instead of making something up.

This is exactly the principle validated by the founding RAG paper: Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020 — see on Google Scholar), show that pairing a generator with an index of retrievable passages produces answers that are more factual — and updatable by changing the index, without touching the model. That's precisely what your weekly re-crawl does: the chatbot's knowledge updates along with the site, with no retraining of anything.

One refinement is worth it as soon as your visitors search for exact terms — product references, proper nouns, error codes: pure vector search can miss "GX-450" in favor of passages that are semantically close but off-topic. Hybrid search (vector + full-text) fixes this; our hybrid search guide covers setting it up on the same Supabase table. And before announcing the chatbot internally or to customers, put it on the test bench with a set of questions whose answers you know — our guide on evaluating RAG quality gives you the method.

Step 6 — Embedding it on the site

All that's left is putting the chatbot in front of visitors. n8n's Chat Trigger can be exposed publicly and embedded on the site via the chat widget — appearance, welcome messages, position on the page: it's all covered in our guide to the AI chat widget on your website. For internal use (onboarding help, sales support), n8n's built-in chat interface is enough, with nothing exposed publicly.

Common pitfalls

• Crawling without exclusions: legal pages, cart pages, and paginated archives end up in the index, pollute the results, and burn credits for nothing.
• Losing the URL along the way: chunking that doesn't propagate metadata produces a chatbot that can't cite its sources — and is therefore unverifiable.
• Splitting into arbitrary blocks markdown that's already structured by headings: answers end up cut across two chunks, and relevance drops.
• Reindexing the whole site on every update instead of a per-URL upsert with hash detection: embedding costs multiplied for nothing.
• Updating chunk by chunk: as soon as one paragraph changes, the whole page's splitting shifts; delete and reinsert per URL.
• Forgetting the anti-hallucination instruction in the system message: a public chatbot that invents an answer about your pricing costs more than no chatbot at all.
• Launching without evaluating: test with questions the site covers and questions it doesn't before opening it up to visitors.

In summary

The complete pipeline fits in two n8n workflows: ingestion (scoped Firecrawl crawl → section-based chunking with URL and title in the metadata → embeddings → Supabase pgvector) and conversation (Chat Trigger → agent with vector — or hybrid — search and URL citations), plus a third for the weekly update via upsert and hashing. Each brick is simple; it's the assembly — scope, metadata, update strategy — that makes the difference between a gimmick and an assistant you can put in front of customers. To start from an already-wired base rather than a blank page, the RAG Assistant Pack (€119) provides the ingestion, vectorization, and citations-enabled chatbot workflows ready to import: all that's left is plugging the Firecrawl crawl in as the input and pasting in your keys.