Connecting Apache Kafka to n8n: consuming and publishing events

A Kafka platform often feels like a motorway with no exit: the cluster carries business events — orders created, payments settled, application alerts — but turning one of them into a concrete action means writing a Java or Python consumer, packaging it, deploying it. n8n shifts that balance: two native nodes, Kafka to publish and Kafka Trigger to consume, are enough to plug a business automation onto an existing topic. This guide walks through their actual parameters, the critical question of consumer groups and offsets, and the throughput ceiling you need to know before promising anything real-time.

Kafka is not a message queue

That is the founding misunderstanding, and it shapes everything else. A RabbitMQ queue is a pipe: a message goes in, a consumer takes it out, it disappears. A Kafka topic is an ordered, partitioned, persistent log: messages are appended, retained according to a retention policy, and read without ever being destroyed. The read position therefore belongs to the reader — each consumer group keeps an offset per partition — and replay is native. One last structural point: ordering is only guaranteed within a partition.

The reference study on this comparison remains Philippe Dobbelaere and Kyumars Sheykh Esmaili's Kafka versus RabbitMQ: A comparative study of two industry reference publish/subscribe implementations, presented at the ACM DEBS 2017 conference (see on Google Scholar). RabbitMQ shines there on rich routing and low per-message latency; Kafka is built for massive streams that can be re-read from any point in time. To decouple a webhook from slow processing, RabbitMQ remains the most direct choice; Kafka earns its place when the topic already exists and serves several consumers.

Creating the Kafka credential

n8n uses a single Kafka credential, shared by the producer node and the trigger. Four fields: Client ID, the client identifier as the broker will see it; Brokers, a comma-separated list in <host>:<port> form (kafka-1:9092,kafka-2:9092); SSL, to leave on for any remote cluster; and Authentication, a toggle that reveals Username, Password and SASL Mechanism — three options: Plain, scram-sha-256 and scram-sha-512.

For a managed cluster (Confluent Cloud, Redpanda Cloud, Aiven), the recipe is almost always the same: SSL and Authentication on, Plain or scram-sha-256 depending on the provider, the API key / API secret pair in Username and Password, and the console's bootstrap server in Brokers — all encrypted like any other credential (see the guide to securing your API credentials and the one on environment variables). One caveat: it authenticates the brokers only. To decode Avro through a protected Confluent Schema Registry, n8n provides a separate Schema Registry credential.

Publishing to a topic: the Kafka node

The Kafka node has a single operation, Send message:

• Topic: the destination topic;
• Send Input Data: on by default, it sends the current item's JSON as-is. Turn it off to take control through the Message field, which accepts an expression;
• Use Key then Key: the message key. This is not cosmetic — it determines the partition, and therefore relative ordering. Publishing every event of one order with Key = {{ $json.order_id }} guarantees they land in the same partition;
• Headers, key/value pairs sent as message headers (or Headers (JSON) if you enable JSON Parameters);
• Use Schema Registry, which reveals Schema Registry URL and Event Name — the latter expects the schema as namespace.name.

Three options round it off: Acks (wait for acknowledgement from all replicas), Compression (GZIP) and Timeout, at 30,000 ms by default.

The obvious use case: decoupling slow processing. The workflow behind a webhook validates the request, publishes an event and answers immediately; a second one consumes the topic at its own pace. Same logic as splitting into sub-workflows, but with a durable boundary.

Consuming a topic: the Kafka Trigger node

Only two required fields, Topic and Group ID. Everything else lives under Options, and several defaults deserve a careful read:

• Read Messages From Beginning is on by default: on a long-retention topic, the first publish replays the whole available history;
• JSON Parse Message turns the body into a usable object; Only Message (visible only when parsing is on) returns just the payload; Return Headers adds the Kafka headers to the output;
• Session Timeout is 30,000 ms and Heartbeat Interval 10,000 ms: the broker evicts from the group any consumer whose session expires;
• Max Number of Requests caps unacknowledged requests on a single connection (1 by default) and Partitions Consumed Concurrently the number of partitions handled in parallel (0, meaning sequentially);
• Allow Topic Creation, Auto Commit Threshold and Auto Commit Interval complete the fine tuning.

Without parsing, the output looks like this:

{
  "message": "{\"order_id\":\"A-4182\",\"amount\":149.9}",
  "topic": "orders.created"
}

With JSON Parse Message and Only Message, you get { "order_id": "A-4182", "amount": 149.9 } straight into $json.

Consumer group and offsets: the question that decides everything

Group ID is by far the most consequential parameter. Kafka splits a topic's partitions across the members of one group: if two n8n triggers share a Group ID, they do not receive the same stream, they share the stream. A test workflow published with the production Group ID would silently capture a slice of real traffic, triggering a rebalance along the way. The rule: one Group ID per use case and per environment (n8n-crm-prod, n8n-crm-dev).

Then comes the real question: what happens if the workflow fails after receiving a message? That is the job of the Resolve Offset parameter, with four values:

1. On Execution Completion (default) — the offset advances when the execution ends, whatever its status: a workflow that crashes still consumes the message, and the event is lost.
2. On Execution Success — the offset only advances on success. The message will be re-read, and potentially processed twice.
3. On Allowed Execution Statuses — a finer variant, with an Allowed Statuses list (success, error, crashed…).
4. Immediately — the offset advances on receipt. The documentation explicitly advises against it: an open door to message loss.

Choosing On Execution Success puts you in at-least-once semantics: no event lost, but duplicates are possible. The practical answer is always the same: make the processing idempotent. An event identifier stored in a tracking table before the expensive call, exactly as when deduplicating replayed webhooks, is enough to absorb the replay. An Error Workflow completes the setup, because Kafka offers no native dead letter queue: republishing unrecoverable messages to a *.dlq topic is on you.

Throughput: n8n is not a high-performance consumer

Guenter Hesse, Christoph Matthies and Matthias Uflacker, in How Fast Can We Insert? An Empirical Performance Evaluation of Apache Kafka (IEEE ICPADS 2020 — see on Google Scholar), measure an ingestion rate of roughly 420,000 messages per second on commodity hardware — on a single topic, a single partition and without replication, so a laboratory ceiling rather than a production figure. An n8n workflow calling an LLM handles a handful per second: the gap is four orders of magnitude, and no setting will close it.

Three strategies follow. Filter upstream: do not consume events.all just to discard 99% of it with a Filter node, ask the data team for a pre-filtered topic (orders.created.high_value). Add consumers: n8n's queue mode with Redis spreads executions across several workers, with real parallelism still capped by the number of partitions. Accept the lag: on a chatty topic the consumer falls behind and never catches up — better to monitor it than to discover it three weeks later.

Three patterns that work

• orders.created → AI enrichment → CRM: the trigger reads each order, a model categorises the customer, a HubSpot or Pipedrive node writes the result back. Moderate volume, high added value: the ideal case.
• A bridge to human channels: a topic of technical events filtered on severity, routed to Slack or Teams with a summary generated on the fly.
• Buffering into a database: the trigger writes each message into PostgreSQL and a scheduled workflow aggregates periodically. Conversely, n8n publishes the result of a long job — invoice extraction, transcription — onto a topic other systems will consume.

The traps to know about

• The broker unreachable from the container. A misconfigured advertised.listeners is failure number one: the broker answers the bootstrap call, then returns an address (localhost:9092) that the n8n container cannot resolve. Inside a Docker network it must advertise its service name — the same family of problem as the ECONNREFUSED on localhost error.
• Read Messages From Beginning left at its default. Publishing the workflow then replays the entire history: thousands of executions, billed API calls, duplicates in the CRM.
• Exotic compression. Version 1 of the Kafka Trigger reads uncompressed and GZIP messages; it fails on LZ4, Snappy and Zstd, which are common on JVM and Confluent producers. Either ask the producer to switch to gzip, or move to version 2 of the node (a preview at the time of writing).
• The message that is not JSON. An Avro topic returns unreadable binary: enable Use Schema Registry and add the matching credential.
• The session that expires. Processing longer than Session Timeout (30 s by default) pushes the consumer out of the group, triggers a rebalance, and the message gets redistributed — hence processed twice.
• Allow Topic Creation switched on by mistake. A typo in the topic name then creates an empty topic instead of raising an error, and the workflow waits forever.

Key takeaways

Kafka gives n8n what no REST API can: a persistent, replayable, already-normalised stream of business events. Three parameters decide the real behaviour — Group ID (who shares what), Resolve Offset (what happens on failure) and Read Messages From Beginning (fresh start or full replay). The rest is a matter of scale: n8n is an excellent last mile, not a stream processing engine. Filter upstream, make the processing idempotent, watch the lag — and for everything else, apply the usual workflow performance tuning.

Going further

Plugging a Kafka topic into an automation only pays off if the downstream processing is worth the detour. The RAG Assistant Pack (€119) turns a stream of documents into a queryable knowledge base, with the idempotency discipline that avoids indexing the same event twice. For a topic of alerts or inbound requests, the AI Inbox Pack (€79) provides the triage and drafting block to place right after the Kafka Trigger.