API Pagination in n8n: Looping Cleanly Over Large Data Sets

One need keeps coming up in nearly every workflow that consumes an external API: fetching an entire list — every contact in a CRM, every order in a store, every ticket in a support queue — while the API never returns more than 20, 50, or 100 results per call. The temptation is to test with the first page, see it work, then discover in production that only the first 50 contacts ever got synced. This guide covers how to handle pagination cleanly in n8n, from the HTTP Request node's built-in option to manual loops for the trickier cases.

The three ways an API paginates its results

Before configuring anything, identify the mechanism the target API exposes — it drives everything that follows:

• Page number or offset pagination (?page=2 or ?offset=50&limit=50): the simplest, but fragile if data changes between calls (an item inserted mid-run shifts everything after it).
• Cursor or token pagination (?cursor=eyJpZCI6MTIzfQ or a next_token field in the response): more resilient to concurrent inserts, used by Stripe, HubSpot, or the Notion API.
• Full-URL pagination (a next field or a Link header that already contains the complete next-page URL, as on GitHub): the easiest to follow, since there's nothing to reconstruct yourself.

The pagination type directly determines which setting to pick in the HTTP Request node.

The native method: the HTTP Request node's Pagination option

In the HTTP Request node, the Options tab exposes a Pagination entry that covers most cases without writing a single line of loop logic:

• Update a Parameter in Each Request: n8n automatically increments a parameter (page, offset) on each call. The expression {{ $pageCount + 1 }} references the current iteration number — useful for APIs that paginate starting at 1 while $pageCount starts at 0.
• Response Contains Next URL: n8n follows the next-page URL returned directly in the response (a JSON field or a Link header), without you having to reconstruct the URL yourself.

Either way, a stop condition is required (a max page count, or an expression like "the response's items array is empty"): without it, the node keeps querying the API indefinitely, which ends in a rate-limit error or a run that never finishes.

This setting covers the large majority of classic REST integrations, and it's the first option to try before reaching for something more complex — in the same spirit as choosing between Split In Batches and a simple loop: start with the built-in tool before building a custom workaround.

When native pagination isn't enough: the manual loop

Some APIs don't fit either of those two cases: a short-lived auth token that needs refreshing between pages, a dynamic pause based on an X-RateLimit-Remaining header, or a need to process (write to a database, send a notification) each page as it arrives rather than waiting for the full fetch to finish. In these cases, the classic pattern combines three nodes:

1. An HTTP Request node that fetches one page.
2. An IF node that tests the stop condition (empty cursor, page beyond the total the API reports, empty results array).
3. A Loop Over Items node (formerly Split In Batches) or a loop back into the HTTP Request node itself, as long as the stop condition isn't met.

The advantage over native pagination: each page can be processed immediately (Supabase insert, duplicate check, notification) instead of accumulating every result in memory until the end. This is the setup to favor once volume exceeds a few hundred items, so a giant run doesn't fail twenty minutes in without having persisted anything.

Concrete example: syncing every contact from a CRM

In a workflow similar to those in the RAG Assistant Pack ($119) for Notion syncing, or a HubSpot/Pipedrive CRM sync, the typical loop looks like: call the /contacts endpoint with the current cursor, insert each returned contact into a Supabase table (with an upsert on the external ID to avoid duplicates), then call again with the next_cursor returned by the API — until that field comes back empty. The same pattern applies to fetching orders in an e-commerce workflow or support tickets for AI-based scoring.

Common pitfalls

• Rate limiting: pagination that's too aggressive (no delay between calls) quickly triggers 429 errors on APIs with tight quotas. See our guide on handling 429 errors for backoff strategies to combine with a pagination loop.
• Run memory: accumulating tens of thousands of items in a single array before processing them in bulk can slow the run down significantly, or make it fail outright on a resource-constrained instance. Writing as you go (page by page) instead of at the end solves the problem in nearly all cases.
• Duplicates after a retry: if the workflow fails on page 40 of 100 and starts over from scratch on the next trigger, the first 39 pages get reprocessed. An Error Workflow combined with a tracking table (the last cursor processed, stored for instance in an n8n Data Table or Supabase) lets you resume exactly where the run left off instead of starting over.
• Poorly defined stop conditions: checking items.length > 0 works for most APIs, but some return an empty array before the actual last page, or conversely keep returning a non-null cursor after the data logically ends. Always verify the API's real end-of-pagination behavior rather than relying solely on its documentation.

Why automate rather than paginate by hand

Before automating this step, the alternative is still to click "next page" manually in a web interface and copy the data into a spreadsheet — a practice still common for small, one-off volumes. A study by Barchard and Pace (Preventing human error: The impact of data entry methods on data accuracy and statistical results, Computers in Human Behavior, 2011 — see on Google Scholar) found that simple visual checking of manually entered data produces 29 to 58% more errors than controlled double entry, and that these errors can meaningfully change downstream results. A properly bounded and tested pagination loop eliminates this class of error by construction: every page is handled identically, without fatigue or oversight, from the first call to the last.

Wrapping up

Pagination is an almost invisible building block once it's set up correctly, but it determines whether any workflow that depends on a complete list — rather than a partial glimpse — can be trusted. Start with the HTTP Request node's native Pagination option, only move to a manual loop with Loop Over Items when the use case genuinely requires it, and always process pages as they arrive rather than in bulk: these three habits avoid most of the unpleasant surprises. The workflows in FlowKit's packs apply this principle consistently everywhere they query an external API at volume — from the Inbox AI Pack ($79) to the Compliance & Audit Pack ($149).