Scraping a website with n8n and structuring the data with AI: the complete guide

Monitoring the prices your distributors display, aggregating job postings from several career sites, tracking mentions of your brand on sites that expose no feed, checking whether a product is back in stock: all of these boil down to "read a web page regularly and pull out three pieces of information". Done by hand, it's tedious and quickly abandoned. With n8n, it fits into a few nodes — and AI solves web scraping's historical problem along the way: CSS selectors that break with every site redesign. This guide covers the full method, from the legal framework to establish before anything else through to industrialization with storage, alerts, and deduplication.

Before writing a single node: legality and ethics

Scraping is not a lawless zone, and this is the first point to address — not the last. An academic tutorial by Krotov, Johnson, and Silva, "Tutorial: Legality and Ethics of Web Scraping", published in 2020 in Communications of the Association for Information Systems, proposes exactly that: a legal and ethical analysis framework to walk through before any scraping project — what does the applicable law say, what do the site's terms say, and what impact does my collection have on the target site and the people involved. In practice, that translates into four reflexes:

• Respect the site's robots.txt file: it states what the publisher accepts being crawled. Ignoring it is the shortest path to getting blocked — and an indefensible position in a dispute.
• Read the site's terms of service. Many explicitly prohibit automated extraction; others tolerate it for reasonable use. Your distributors, with whom you already have a contractual relationship, will often accept price monitoring if you mention it.
• The GDPR applies as soon as personal data appears: recruiter names on a job posting, review authors, email addresses. Publicly accessible data is not freely reusable data — you need a legal basis, a retention period, and a defined purpose.
• Prefer the official API or RSS feed when they exist. It's more stable, faster, and it's the channel the publisher intended for you. Scraping is plan B, not plan A.

Add to that a reasonable pace: one request every few seconds, not fifty in parallel. You want to read a page the way a devoted visitor would, not subject the target server to a load it wasn't sized for. And if a site blocks you or shows a captcha, that's a signal to respect, not an obstacle to work around.

The simple method: HTTP Request + HTML Extract

For a classic server-rendered site, two nodes are enough. An HTTP Request node in GET fetches the page's HTML. An HTML node (operation Extract HTML Content) then applies CSS selectors to pull out the fields you want:

• h1.product-title → product name
• .price → displayed price
• .availability → stock status

Each selector becomes a key in the output JSON, directly usable by the following nodes. To find the right selector, a right-click → "Inspect" in the browser on the target element does the job in most cases. Test with the "Return Array" option enabled when the page lists multiple elements (several job postings, several products): you get an array that a Split Out node turns into individual items.

This method is free, fast, and perfect for pages whose structure rarely moves. Its weakness is well known: the day the site changes its CSS classes, the extraction returns empty fields without raising an error. Hence the value of validating the output (an IF that checks the price isn't empty) and alerting when extraction fails.

The limit: JavaScript-rendered sites

More and more sites build their content in the browser: the HTML returned by HTTP Request contains an empty shell, and the useful content is injected by JavaScript afterwards. Typical symptom: your selectors are correct in the browser's inspector, but the HTML node finds nothing. Two options, in this order:

1. Look for the JSON API the site calls itself. Open the network tab of the developer tools, reload the page, filter on XHR/Fetch: you'll often find a call that returns exactly the data being displayed, already as clean JSON. Call that URL directly from HTTP Request — it's more reliable than any HTML parsing, and if the API is paginated, our guide on API pagination in n8n shows how to fetch everything cleanly.
2. Use an external headless rendering service (Browserless, ScrapingBee, and equivalents): n8n sends the URL to the service, which executes the JavaScript in a real browser and returns the final HTML, which you then process normally. That adds a cost and a dependency — reserve it for the pages that justify it.

What AI brings: replacing fragile selectors with an LLM

This is where the modern approach changes the game. Instead of targeting .price-v2__amount--discounted, send the page's text (the HTML converted to text via a Markdown or HTML node, to save tokens) to an LLM with an extraction instruction and an output schema: "extract the product name, the price in euros, the publication date, the availability; respond only in JSON". A Structured Output Parser node guarantees the response follows the schema — the exact mechanics are covered in our Structured Output Parser guide.

The advantage is decisive for unstable sources: a layout redesign breaks nothing, since the LLM reads the content like a human would, not the DOM structure. The trade-offs are real:

• a cost per page — negligible for fifty pages a day with an economical model, significant at scale;
• a risk of error — the model can misread a struck-through price or confuse two dates. So validate the critical fields: an IF that checks the price is a number within a plausible range, and a quarantine step (rather than a silent insert) when validation fails.

In practice, the winning combination is often hybrid: CSS selectors for the stable sites you know well, an LLM for heterogeneous or fast-changing sources.

Industrializing: pacing, retries, storage, alerts

Useful scraping is scraping that runs on its own. The skeleton of the production workflow:

1. Schedule Trigger to launch the run once to a few times a day — no point polling a pricing page every five minutes when it changes once a week.
2. Loop over the URL list: store your targets in a table rather than hard-coding them, then iterate with a Loop Over Items node, inserting a Wait node of a few seconds between requests — that's the reasonable pace from the first section, applied concretely.
3. Retry on HTTP errors: a target site can return 503 or time out. Enable "Retry On Fail" on the HTTP Request node with a delay between attempts, and handle permanent failures cleanly — our article on HTTP Request retries and timeouts details the right settings.
4. Storage: a Supabase table for serious volumes and SQL queries, or a shared Google Sheets when the team wants to browse the data without another tool.
5. Alert on change: before writing, compare the extracted value to the last stored value. A price moving at a distributor, a product back in stock, a new job posting matching your criteria → immediate Slack message or email. It's the same threshold-based routing reflex as in our AI competitive monitoring pipeline.

Deduplicating between two runs: idempotency

Scheduled scraping revisits the same pages on every run: without a safeguard, your table fills up with duplicates and your alerts fire in a loop. The remedy is the same as for webhooks: give each result a stable key — the listing URL, a product reference, or a hash of source + title — and put a unique constraint on it in the database. Before each insert, a node checks whether the key exists: if so, update the row (and only fire the alert if a monitored value actually changed); otherwise, insert. This idempotency principle, detailed in our article on idempotency and duplicates in n8n, makes the workflow replayable without side effects: a run relaunched after a failure never pollutes the data.

Going further

A robust scraping pipeline with n8n ultimately fits in one sentence: a legal framework checked upfront, HTTP Request + HTML Extract for the simple cases, an LLM with structured output for fast-changing sources, and an industrialization layer — scheduling, pauses, retries, storage, deduplication — that turns the prototype into a reliable tool. These building blocks are exactly the ones we assemble in FlowKit workflows: the structured-output LLM chain and threshold-based routing from the Inbox AI Pack apply as-is to scraped data, and the monitoring pipeline described above is its natural extension. Start with a single URL and three fields, validate extraction quality over a week, then widen the list — that's the shortest path to a collection process that runs without you.