Backend cheatsheetKafka vs RabbitMQPick the right pipe

Kafka vs
RabbitMQ

One keeps a log that readers walk through at their own pace. The other routes each message to the queues that want it. Here is how each works, what each guarantees and how to choose.

Compare the models Go to the decision Key terms in plain words

By Shree Kumar SharmaKafkaRabbitMQ7 min read

Key terms in plain words

New to messaging systems? Start here. Every word below shows up later on this page, and each one comes with an everyday comparison from a post office. Underlined words in the modules link back to these cards.

Passing notes between programs How Kafka keeps things How RabbitMQ routes things Not losing anything

24 terms in 4 groups. Hover an underlined word anywhere on the page for a quick reminder, or click it to jump here.

Passing notes between programs

Both tools are post offices for software. One program drops off a letter, the post office holds it, and another program collects it when it is ready.

Message: Think of it as one letter in the post; A small packet of information one program sends to another, such as "order 42 was paid". It usually carries an id and some data written as text.
Message broker: Think of it as the post office in the middle; A server that takes messages from senders, holds them safely and hands them to receivers. The sender and receiver never have to talk directly or be running at the same time.
Producer: Think of it as the person posting a letter; The program that sends messages. RabbitMQ calls it a publisher; Kafka calls it a producer.
Consumer: Think of it as the person collecting their post; The program that receives messages and does something with them, such as charging a card or updating a report.
Event: Think of it as a notice saying "this has happened"; A message that records a fact in the past, such as "message sent" or "order shipped". An event stream is the never ending list of them, in the order they happened.
Queue: Think of it as a pigeonhole where letters wait for one person; A waiting line of messages. Each message is handed to one consumer, and once that consumer says it is done the message is deleted.

How Kafka keeps things

Kafka is the post office's record book. Every letter is written in and stays there, and each reader keeps their own bookmark.

Log: Think of it as a record book where new entries only ever go at the end; An ordered list of messages that are added but never changed. Reading a message does not remove it, so many readers can go through the same log at their own pace.
Topic: Think of it as one record book for one subject, like "payments"; A named stream of messages about one kind of thing. Producers write to a topic and consumers read from it.; For example messages.sent is a topic.
Partition: Think of it as splitting one record book into several volumes so several clerks can read at once; A topic is divided into partitions, and each one is its own ordered log. More partitions let more consumers work in parallel, but order is only kept inside one partition.
Key: Think of it as the name on the envelope that decides which volume it goes in; A label on a Kafka message, such as a user id. Messages with the same key always land in the same partition, which keeps one user's events in order.
Offset: Think of it as a bookmark showing how far you have read; The position number of a message in a partition. A consumer group saves the offset it has reached, called committing, so after a restart it carries on from there.
Consumer group: Think of it as a team of clerks sharing the volumes between them; A set of consumers that split a topic's partitions, so each message is handled once per group. Two different groups each read everything, independently.
Retention and replay: Think of it as keeping old record books on the shelf so you can read them again; Kafka keeps messages for a set time even after they are read. A new or fixed consumer can go back and read the history again, which is called replay.

How RabbitMQ routes things

RabbitMQ is the sorting room. Letters come in, sorting rules copy them into the right pigeonholes, and each one is thrown away once it is collected.

Exchange: Think of it as the sorting room where letters arrive; Where publishers send messages in RabbitMQ. The exchange never stores anything itself; it decides which queues get a copy. Its type (direct, topic, fanout or headers) sets how it decides.
Binding: Think of it as a sorting rule pinned to a pigeonhole: "anything for Pune comes here"; A link from an exchange to a queue, with a pattern that says which messages the queue wants. In a topic binding, * matches exactly one word and # matches any number.
Routing key: Think of it as the address written on the envelope; A short dotted label on a RabbitMQ message that the exchange compares with each binding's pattern.; For example send.whatsapp.in is a routing key.
Ack: Think of it as signing for a parcel so the post office can close the case; Short for acknowledgement. The consumer tells the broker "done", and only then is the message deleted. A nack says "failed", and the message is retried or set aside.
Dead letter exchange: Think of it as the post office's undeliverable mail room; Where RabbitMQ sends messages that failed or were rejected, so they don't vanish and don't block the queue. You can inspect them later and send them again.

Not losing anything

How both systems make sure every letter arrives, even when a machine breaks halfway.

At least once: Think of it as a courier who would rather deliver twice than not at all; A promise that no message is lost, at the cost that some may arrive twice, for example after a retry. Both Kafka and RabbitMQ work this way by default when set up safely.
Idempotent: Think of it as a stamp that marks a letter "already handled", so a second copy is ignored; Doing something twice has the same effect as doing it once. An idempotent consumer checks whether it has seen a message's id before, so duplicates do no harm.
Replication: Think of it as keeping carbon copies of every letter in another branch; Storing each message on several servers, so losing one machine loses nothing. In Kafka that means replicated partitions; in RabbitMQ, quorum queues.
Durable and persistent: Think of it as filing letters in a fireproof cabinet rather than on the counter; A durable queue survives a restart of the broker; a persistent message is written to disk rather than held only in memory. You need both to survive a crash.
Publisher confirms: Think of it as a receipt from the counter that your letter was accepted; The broker tells the sender each message is safely stored. Until the confirm arrives, the sender should treat the message as not sent yet.
Transaction: Think of it as a deal where either every step goes through or none does; A group of changes that succeed together or are all undone together. Saving a message's id and doing the work in one database transaction is how a consumer avoids doing it twice.

A log or a broker

Kafka keeps an ordered log that many readers walk through at their own pace. RabbitMQ routes each message to queues and deletes it once handled.

ConceptsMust knowLogQueue

Kafka: a logappend(offset 6)Messages stay for the retention period. Each consumer group remembers where it is and can rewind.

RabbitMQ: a brokerroute(order.created.in)Exchanges copy messages into matching queues. An acked message is gone.

	Kafka	RabbitMQ
Unit	Topic split into partitions	Exchange, bindings, queues
After a consumer reads	Message stays; the group moves its offset	Message is removed on ack
Many consumers of one event	Each group reads everything	Bind one queue per consumer type
Ordering	Per partition, by key	Per queue, with one consumer
Who tracks progress	The consumer group (offsets)	The broker (acks)
Sweet spot	Event streams, history, analytics	Tasks, routing, request and reply

Kafka in one picture

Topics, partitions, keys, consumer groups and offsets. Five words that explain almost everything Kafka does.

KafkaPartitionsConsumer groupsOffsets

Picture it: one topic, two consumer groups

Messages with the same key always land in the same partition, which is what keeps one user’s events in order.

Inside a group, each partition is read by one consumer at a time, so a group can never use more consumers than there are partitions. Run this to see how six partitions are shared as a group grows.

assign.jsJavaScript

const partitions = [0, 1, 2, 3, 4, 5];

function rangeAssign(consumers) {
  const per = Math.floor(partitions.length / consumers);
  const extra = partitions.length % consumers;
  let next = 0;
  return Array.from({ length: consumers }, (_, i) => {
    const take = per + (i < extra ? 1 : 0);
    const mine = partitions.slice(next, next + take);
    next += take;
    return 'c' + (i + 1) + '=[' + mine.join(',') + ']';
  }).join(' ');
}

for (const n of [1, 2, 4, 6, 8]) console.log(n + ' consumers: ' + rangeAssign(n));

TerminalOutput

$ node assign.js
1 consumers: c1=[0,1,2,3,4,5]
2 consumers: c1=[0,1,2] c2=[3,4,5]
4 consumers: c1=[0,1] c2=[2,3] c3=[4] c4=[5]
6 consumers: c1=[0] c2=[1] c3=[2] c4=[3] c5=[4] c6=[5]
8 consumers: c1=[0] c2=[1] c3=[2] c4=[3] c5=[4] c6=[5] c7=[] c8=[]

Why it matters: past six consumers the extras sit idle, so choose the partition count for the parallelism you will need.

key'u_91'

Same key, same partition, strict order for that key.

See the example

Ordering

partitions: 6

The ceiling on parallel consumers per group.

See the example

Scale

retention.ms

How long messages stay, read or not. Days, or forever with compaction.

See the example

History

lag

Latest offset minus committed offset. The one Kafka metric to alert on.

See the example

Health

RabbitMQ in one picture

Publishers send to exchanges, exchanges copy into queues by binding rules, and consumers ack each message they finish.

RabbitMQExchangesBindingsAcks

Picture it: one message, three bindings

Exchange type	Routes by	Use for
direct	Exact routing key	Work queues per task type
topic	Key patterns with * and #	Events by region, type or tenant
fanout	Nothing, copies to every bound queue	Broadcasts, cache invalidation
headers	Message header values	Rare; when keys are not enough

In a topic binding, * matches exactly one word and # matches zero or more. Run this to check a few keys against the bindings above.

topic-match.jsJavaScript

function matches(pattern, key) {
  const p = pattern.split('.'), k = key.split('.');
  const walk = (i, j) => {
    if (i === p.length) return j === k.length;
    if (p[i] === '#') return walk(i + 1, j) || (j < k.length && walk(i, j + 1));
    return j < k.length && (p[i] === '*' || p[i] === k[j]) && walk(i + 1, j + 1);
  };
  return walk(0, 0);
}

const bindings = { billing: 'order.*.in', audit: '#', shipping: 'order.shipped.*' };
for (const key of ['order.created.in', 'order.shipped.in', 'order.created.us.east']) {
  const to = Object.keys(bindings).filter((q) => matches(bindings[q], key));
  console.log(key.padEnd(22) + ' -> ' + to.join(', '));
}

TerminalOutput

$ node topic-match.js
order.created.in       -> billing, audit
order.shipped.in       -> billing, audit, shipping
order.created.us.east  -> audit

Why it matters: routing rules live in the broker, so adding a consumer is a new binding, not a code change in the publisher.

Delivery and ordering

Both can lose nothing and both will sometimes deliver twice. Design consumers to be idempotent either way.

ReliabilityAt least onceIdempotencyOrdering

Guarantee	Kafka	RabbitMQ
Publisher knows it landed	`acks=all` with enough in sync replicas	Publisher confirms
Survives a broker restart	Replicated partitions	Durable quorum queues and persistent messages
At least once to consumers	Commit offsets after processing	Manual ack after processing
Duplicates from producer retries	Idempotent producer removes them	Possible; dedupe in the consumer
Exactly once	Transactions for read, process, write inside Kafka	Not offered; use idempotent consumers
Order kept	Within a partition	Within a queue, with one consumer and no requeue

Receivemessage plus its id
Seen before?check a processed ids table
Do the workin one DB transaction
Record the idsame transaction
Ack or commitonly now

The code, side by side

A producer and a consumer for each, with the settings that make them safe by default.

Node.jsKafkaJSamqplibAcks

kafka.tsTypeScript

const kafka = new Kafka({ clientId: 'bridgeq', brokers: process.env.KAFKA_BROKERS!.split(',') });

const producer = kafka.producer({ idempotent: true });
await producer.connect();
await producer.send({ topic: 'messages.sent', acks: -1, messages: [{ key: tenantId, value: JSON.stringify(event) }] });

const consumer = kafka.consumer({ groupId: 'analytics' });
await consumer.subscribe({ topic: 'messages.sent', fromBeginning: false });
await consumer.run({
  eachMessage: async ({ partition, message }) => {
    await analytics.record(JSON.parse(message.value!.toString())); // offset commits after this resolves
  },
});

Why it matters: keying by tenant keeps each tenant’s events in order, and idempotent sends stop retry duplicates. For new projects also look at Confluent’s JavaScript client, which offers a KafkaJS compatible API.

rabbit.tsTypeScript

const conn = await amqp.connect(process.env.AMQP_URL!);
const ch = await conn.createConfirmChannel();
await ch.assertExchange('messages', 'topic', { durable: true });
await ch.assertQueue('whatsapp.send', { durable: true, arguments: { 'x-queue-type': 'quorum', 'x-dead-letter-exchange': 'messages.dlx' } });
await ch.bindQueue('whatsapp.send', 'messages', 'send.whatsapp.*');

ch.publish('messages', 'send.whatsapp.in', Buffer.from(JSON.stringify(msg)), { persistent: true, messageId: msg.id });
await ch.waitForConfirms();

await ch.prefetch(20); // at most 20 unacked messages per consumer
await ch.consume('whatsapp.send', async (m) => {
  try { await whatsapp.send(JSON.parse(m!.content.toString())); ch.ack(m!); }
  catch { ch.nack(m!, false, false); } // to the dead letter exchange, not back to the queue
});

Why it matters: confirms prove the broker has it, prefetch keeps one slow consumer from hoarding, and failures park in a dead letter queue instead of looping.

Choose, then combine

Start from what the consumers need. Many systems end up with both: Kafka for the event stream, RabbitMQ for routing work.

DecisionMust knowTrade offsBoth

Picture it: three questions

BridgeQ eventsmessage.sentdeliveredread kept in Kafka, replayed by analytics

BridgeQ routingsend.whatsapp.*send.sms.*send.email.* routed by RabbitMQ to channel workers

If you need	Lean to
Replay, audit trails, event sourcing	Kafka
Many independent readers of one stream	Kafka
Very high sustained throughput	Kafka
Routing by pattern, per task queues	RabbitMQ
Per message acks, retries and dead letters	RabbitMQ
Priorities, TTL per message, request and reply	RabbitMQ
Simple jobs inside one Node.js service	Neither: BullMQ on Redis

Which one do I need?

Name the job, then pick the tool. When the answer is both, give each the half it is good at.

I want to	Use	Because
Keep every event and replay it later	Kafka	Messages stay for the retention period
Let five teams read the same stream	Kafka	Each consumer group has its own offsets
Route tasks by type or tenant	RabbitMQ	Topic exchanges and bindings do it in the broker
Retry one failed message, then park it	RabbitMQ	Per message nack and dead letter exchanges
Keep one user’s events in order	Kafka	Same key, same partition
Process each message exactly once	Either, with idempotent consumers	Duplicates happen on retries
Queue background jobs in one app	BullMQ	Less to run than either broker