Apache Kafka is a distributed streaming platform designed to handle high throughput, real-time data pipelines, and event-driven applications. One of its core features is partitioning, which allows Kafka topics to be divided into multiple partitions for scalability, fault tolerance, and parallel processing. Understanding how messages are delivered across these partitions is critical for designing efficient and predictable Kafka-based systems. This article explores how message delivery works with multiple partitions.
1. What Are Partitions in Kafka?
A Kafka topic is a logical channel where producers publish messages and consumers read them. To enable scalability and fault tolerance, each topic is divided into partitions, which are the smallest unit of parallelism in Kafka.
A partition is essentially an ordered, immutable log where messages are appended sequentially and assigned a unique incremental offset. For instance, a topic such as orders could be split into three partitions, each one maintaining its own independent sequence of messages.
Each partition is stored on a broker (with possible replicas on others), allowing Kafka to scale horizontally by distributing data across multiple machines. Producers write messages to partitions based on a partitioning strategy (round-robin, key-based, or custom), while consumers in a group are assigned specific partitions to process in parallel.
The choice of partitioning strategy directly impacts message ordering and load balancing. Ordering is preserved only within a partition, not across the entire topic, which makes understanding partitions essential for designing efficient Kafka systems.
2. Message Delivery With a Key (Hashing Strategy)
When you use a key in Kafka messages, the producer applies a hashing algorithm (MurmurHash2) on the key to decide the partition. This ensures that all messages with the same key are directed to the same partition, thereby preserving the ordering for that key.
Example: Producer with Key-Based Partitioning
public class KeyBasedProducer {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
String topic = "multi-partition-topic";
// Messages with the same key always go to the same partition
producer.send(new ProducerRecord<>(topic, "Customer-A", "Order-101"));
producer.send(new ProducerRecord<>(topic, "Customer-A", "Order-102"));
producer.send(new ProducerRecord<>(topic, "Customer-B", "Order-201"));
producer.send(new ProducerRecord<>(topic, "Customer-B", "Order-202"));
producer.send(new ProducerRecord<>(topic, "Customer-C", "Order-301"));
producer.close();
}
}
In this example, the producer sends records with customer IDs as keys. Kafka uses the key’s hash to map each message consistently to the same partition. As a result:
- All messages with key
Customer-Ago to the same partition and preserve ordering (Order-101 -> Order-102). - All messages with key
Customer-Bgo to another partition and preserve ordering (Order-201 -> Order-202).
This approach is beneficial when message order matters for a subset of data, such as ensuring all events for a specific customer, account, or session are processed sequentially. At the same time, using multiple partitions allows Kafka to scale horizontally since messages with different keys can be processed in parallel across partitions.
3. Message Delivery Without a Key (Round-Robin Strategy)
When messages are sent without a key, Kafka distributes them across partitions in a round-robin manner. This ensures that partitions receive messages evenly, which improves throughput and load balancing. However, because messages are scattered across different partitions, ordering is not guaranteed globally—only within each partition.
Example: Producer with Round-Robin for Keyless Messages
public class RoundRobinProducer {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
String topic = "roundrobin-topic";
// Sending messages without a key
for (int i = 1; i <= 10; i++) {
producer.send(new ProducerRecord<>(topic, null, "Message-" + i));
}
producer.close();
}
}
In this example, each message is created with a null key, which triggers Kafka’s round-robin strategy, distributing messages evenly across all available partitions. For instance, with three partitions, the producer assigns Message-1 to partition 0, Message-2 to partition 1, Message-3 to partition 2, then loops back to assign Message-4 to partition 0, and so forth.
While this approach balances load efficiently, it does not preserve the overall order of messages since consecutive messages may land in different partitions, making it best suited for stateless or independent events such as logging, telemetry, or sensor data.
4. Ordering Guarantees Across Partitions
One important aspect of Kafka’s design is how it handles the ordering of messages. Kafka provides strict ordering guarantees, but these guarantees only apply within a single partition. Across multiple partitions, no ordering is guaranteed.
This means that when all related messages are directed to the same partition using a key, their order is preserved, but if messages are distributed across multiple partitions, such as through round-robin without a key, consumers may read them in a different sequence than the producer originally sent.
Example Without a Key (Ordering Lost Across Partitions)
In this example, we create a producer that sends four simple order messages without specifying any key. Since no key is provided, Kafka uses a round-robin strategy to distribute them across partitions.
public class OrderingWithoutKeyExample {
public static void main(String[] args) {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
Producer<String, String> producer = new KafkaProducer<>(props);
for (int i = 1; i <= 4; i++) {
producer.send(new ProducerRecord<>("orders", "Order-" + i));
}
producer.close();
}
}
The producer sends the messages in the order: Order-1, Order-2, Order-3, and Order-4.
However, because Kafka distributes them across partitions, they may not be consumed in this same order. For instance, a consumer might read Order-2 first, followed by Order-3, then Order-1, and finally Order-4. This happens because ordering is only guaranteed within each individual partition, not across the entire topic.
Example With a Key (Ordering Preserved)
In this example, we send messages with a key, such as a customer ID. This guarantees ordering for those messages.
public class OrderingWithKeyExample {
public static void main(String[] args) {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
Producer<String, String> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("orders", "customer1", "Order-1"));
producer.send(new ProducerRecord<>("orders", "customer1", "Order-2"));
producer.send(new ProducerRecord<>("orders", "customer1", "Order-3"));
producer.close();
}
}
Here, all three orders for the same customer are routed to the same partition. As a result, a consumer will always read them in the exact sequence they were sent: Order-1, followed by Order-2, and then Order-3. This is how Kafka preserves per-key ordering while still allowing for scalability across partitions.
5. Delivery Guarantees
Beyond delivery, Kafka also provides processing guarantees that define how consumers handle messages: At-most-once means messages may be lost but will never be processed again; At-least-once ensures that no messages are lost, although some may be processed more than once; and Exactly-once guarantees that every message is processed once and only once.
Example: At-least-once Processing
while (true) {
for (ConsumerRecord<String, String> record : consumer.poll(1000)) {
processOrder(record.value()); // process business logic
consumer.commitSync(); // commit after processing
}
}
If the consumer crashes after processing but before committing, Kafka will redeliver the message, which can result in duplicate processing but ensures no data loss.
6. Rebalancing and Consumer Group Dynamics
Consumer groups automatically adapt when consumers join or leave. This process is called rebalancing. If a new consumer joins, partitions are redistributed among group members. If a consumer crashes, its partitions are reassigned to the remaining consumers.
For example, consider a topic named orders with six partitions. In a consumer group with three members, each consumer is assigned two partitions. If one consumer leaves, the remaining two consumers will each take over three partitions. Rebalancing ensures fault tolerance but introduces brief downtime while partitions are reassigned.
public class ConsumerWithRebalance {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-consumer-group");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false"); // we commit manually
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
// Handle rebalance events
consumer.subscribe(Arrays.asList("orders"), new ConsumerRebalanceListener() {
@Override
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
System.out.println("Partitions revoked: " + partitions);
// Commit offsets before rebalance so no messages are lost
consumer.commitSync();
}
@Override
public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
System.out.println("Partitions assigned: " + partitions);
}
});
try {
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> record : records) {
System.out.printf("Consumer %s processing record: key=%s, value=%s, partition=%d, offset=%d%n",
args.length > 0 ? args[0] : "1",
record.key(), record.value(), record.partition(), record.offset());
}
consumer.commitSync();
}
} finally {
consumer.close();
}
}
}
In this example, when a rebalance occurs, the listener prints which partitions were revoked and reassigned. This ensures smooth partition handover between consumers, though frequent rebalancing can cause temporary downtime.
7. Conclusion
In this article, we explored how Kafka manages message delivery across topics, partitions, and consumer groups, while also covering ordering and rebalancing. We examined key concepts such as partitioning strategies and processing guarantees. The central takeaway is that Kafka message delivery with multiple partitions may occur in any order unless a key is provided to enforce partition-level sequencing.
8. Download the Source Code
This article explored Kafka message delivery across multiple partitions.
You can download the full source code of this example here: kafka message delivery multiple partitions
