Apache Kafka Tutorial: Build a KRaft Cluster in 13 Steps [2026]

Apache Kafka 4.2.0 landed on February 17, 2026, and it changes how Java developers, data engineers, and platform teams design streaming systems in production. This Apache Kafka tutorial walks you through every step needed to install Kafka 4.2, run a KRaft-based cluster without ZooKeeper, build producers and consumers in Java and Python, ship Kafka Streams jobs with the new Dead Letter Queue support, and operate Share Groups (Queues for Kafka) that finally graduated to general availability. By the end, you will have a complete working project on your laptop and a clear path to deploy it to AWS MSK, Confluent Cloud, or Strimzi on Kubernetes.

The tutorial assumes you have used a terminal before but have never run Kafka. Every step is reproducible on macOS, Linux, and Windows Subsystem for Linux. We cover 13 hands-on steps, 5 production code blocks, 8 troubleshooting recipes, and 6 common pitfalls that catch new operators in their first week. Where the docs are thin, we link to the canonical Apache Kafka documentation and the relevant Kafka Improvement Proposal (KIP) so you can dig deeper after the basics click.

·

Google · Preferred Sources

Don't miss new tech stories on Google

Add Tech Insider once in the Google app and our stories appear in your news suggestions.

Add Now

Why Apache Kafka in 2026: The 4.2 Release at a Glance

Apache Kafka has been the de facto event streaming platform for almost a decade, but the gap between the 3.x series and the new 4.x line is larger than any single release before it. The 4.0 release (March 2025) ripped out the ZooKeeper code path entirely and made KRaft the only supported mode. The 4.2.0 release on February 17, 2026 promoted Queues for Kafka (KIP-932) to general availability, shipped the new Streams Rebalance Protocol (KIP-1071), added native Dead Letter Queue support to Kafka Streams (KIP-1034), exposed rack IDs through the Admin API (KIP-1227), and rebuilt the build and test matrix around Java 17 and Java 25.

Practically, that means the cluster you stand up in this Kafka tutorial has no external metadata store, no Apache ZooKeeper ensemble to babysit, no --zookeeper flags anywhere, and a much simpler operational story. It also means the consumer-group rebalance pain that has driven engineers to RabbitMQ and Pulsar for years is being addressed directly on the broker side rather than in the client. If you have been postponing a migration off Kafka 2.x or 3.x, the 4.2 series is the version that justifies the upgrade.

Apache Kafka Release	Date	Headline Feature	Status
3.7.0	February 2024	KRaft GA, ZooKeeper deprecated	Maintenance
3.8.0	July 2024	Tiered Storage GA (KIP-405)	Maintenance
3.9.0	November 2024	Final 3.x line, KRaft migration tooling	Bridge release
4.0.0	March 2025	ZooKeeper removed, KRaft only	Stable
4.1.0	September 2025	Share Groups preview, Streams improvements	Stable
4.2.0	February 17, 2026	Queues GA, Streams Rebalance Protocol, DLQ	Latest

The release rhythm has tightened to roughly two minor versions per year, with patch releases (4.2.1, 4.2.2) shipping inside six weeks when CVEs land. The Kafka security team published advisories for CVE-class issues affecting clients up to 4.1.1 in April 2026, all fixed in 4.2.0, so running an older 4.x build in production now means you are knowingly exposed. Staying current is no longer a “nice to have.”

Prerequisites and Versions for This Apache Kafka Tutorial

Before you run a single command, line up the dependencies. Kafka 4.2 broke compatibility with several older runtimes on purpose. Trying to start it on an unsupported JDK is the single most common reason new tutorials fail in the first ten minutes. Use the exact versions in the table below. They match what Apache Kafka tests against on its CI grid.

Component	Required Version	Why It Matters
JDK (broker side)	Java 17 or Java 25	Kafka 4.2 broker builds against Java 17 minimum, tests on Java 25
JDK (client side)	Java 11+	Clients and Streams retain javac release=11 for backwards compat
Scala (built-in)	2.13.x	Only Scala 2.13 is supported in 4.x
Apache Kafka	4.2.0	Released Feb 17, 2026 from kafka.apache.org/downloads
Operating System	Linux, macOS, WSL2	Native Windows is not officially supported for production
Disk space	10 GB free	Logs grow fast even in dev
RAM	4 GB minimum	Default broker heap is 1 GB, controller wants 512 MB
Python (optional)	3.11+	For confluent-kafka client examples
Docker (optional)	24.x+	For the Strimzi/CP-Server walkthrough

Pick a clean working directory. Across this Kafka tutorial we use ~/kafka-tutorial on Linux/macOS and assume your shell is bash or zsh. If you are on a Mac with Apple Silicon, install a 64-bit ARM JDK build, not Rosetta x86. Running Kafka under Rosetta will work but performance characteristics for benchmarks will be misleading.

Step 1: Install Java 17 or 25 and Verify the JVM

Kafka is a JVM application end to end. The broker, the controller process, and the official Java client all run on a JDK. We recommend Eclipse Temurin (formerly AdoptOpenJDK) because the binaries are maintained by the Adoptium project and are installed by Confluent and Strimzi internally for their CI. Pick Java 25 if you want the latest GC improvements; pick Java 17 if you need to match a long-term-supported production target.

# macOS with Homebrew
brew install --cask temurin@25

# Ubuntu/Debian
sudo apt update
sudo apt install -y openjdk-25-jdk

# Fedora/RHEL
sudo dnf install -y java-25-openjdk-devel

# Verify
java --version
# openjdk 25 2025-09-16
# OpenJDK Runtime Environment Temurin-25+36 (build 25+36)
# OpenJDK 64-Bit Server VM Temurin-25+36 (build 25+36, mixed mode)

Set JAVA_HOME in your shell profile so Kafka’s start scripts can find the JDK without searching $PATH. On macOS that means export JAVA_HOME=$(/usr/libexec/java_home -v 25) in ~/.zshrc. On Linux it is usually export JAVA_HOME=/usr/lib/jvm/java-25-openjdk-amd64. Source the file or open a new shell.

Step 2: Download Apache Kafka 4.2.0 and Inspect the Layout

Apache distributes Kafka as a single tarball that contains the broker, the controller code path, the command-line tools, the Streams library, and Kafka Connect. There is no separate “server” and “client” download. Pull the latest release from the official downloads page and verify the SHA-512 checksum before extracting; the Apache release process publishes the hash next to the binary.

mkdir -p ~/kafka-tutorial && cd ~/kafka-tutorial
curl -O https://downloads.apache.org/kafka/4.2.0/kafka_2.13-4.2.0.tgz
curl -O https://downloads.apache.org/kafka/4.2.0/kafka_2.13-4.2.0.tgz.sha512

# Confirm the checksum
shasum -a 512 -c kafka_2.13-4.2.0.tgz.sha512
# kafka_2.13-4.2.0.tgz: OK

tar -xzf kafka_2.13-4.2.0.tgz
cd kafka_2.13-4.2.0
ls -1
# bin       config    libs      LICENSE   licenses  NOTICE    site-docs

Take a minute to read the directory layout. bin/ holds shell scripts that wrap the JVM calls, config/ holds the property files for every component (broker, controller, connect, streams), libs/ holds the JARs that get added to the classpath, and site-docs/ contains a static HTML mirror of the documentation that ships with the version you downloaded. The shell scripts in bin/ all support a --help flag; you will use it more than you expect.

Step 3: Configure a KRaft Single-Node Cluster

In Kafka 4.x there is no ZooKeeper. Cluster metadata lives inside Kafka itself, replicated through a Raft quorum of controller processes. For learning, we run a “combined” node that acts as both controller and broker on a single JVM. For production, you would split the roles onto dedicated nodes, but the wire protocol and the configuration shape are identical.

Kafka ships with three KRaft sample configs in config/kraft/: server.properties (combined node), broker.properties, and controller.properties. We use the combined config and customize three values: a unique cluster.id, a stable storage directory, and an explicit advertised listener so producers from another machine can connect later.

# Generate a random cluster UUID
KAFKA_CLUSTER_ID=$(bin/kafka-storage.sh random-uuid)
echo $KAFKA_CLUSTER_ID
# k8sP9aQwR0eGv2hNjFvBwQ

# Create a copy of the sample config so future upgrades stay clean
cp config/kraft/server.properties config/kraft/tutorial.properties

# Edit three keys
sed -i.bak 's|^log.dirs=.*|log.dirs=/tmp/kraft-tutorial-logs|' config/kraft/tutorial.properties
sed -i.bak 's|^advertised.listeners=.*|advertised.listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093|' config/kraft/tutorial.properties

# Format the storage directory with the cluster ID
bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/tutorial.properties
# Formatting /tmp/kraft-tutorial-logs with metadata.version 4.2-IV0.

The format command writes a meta.properties file inside log.dirs that records the cluster ID and the metadata version. Once formatted, you cannot reuse the directory for a different cluster ID; that would corrupt the log. If you ever need to wipe and start over, delete the directory and re-run the format step.

Step 4: Start the Kafka Broker and Confirm It Is Alive

The broker is started with bin/kafka-server-start.sh and the path to the property file. By default it logs to stdout and to logs/server.log inside the install directory. Run it in the foreground for the tutorial; in production you would launch it under systemd, supervisord, or the Strimzi Cluster Operator on Kubernetes.

bin/kafka-server-start.sh config/kraft/tutorial.properties

# Expected output (truncated):
# [KafkaRaftServer nodeId=1] Kafka Server started (kafka.server.KafkaRaftServer)
# [BrokerLifecycleManager id=1] The broker has been registered (kafka.server.BrokerLifecycleManager)
# [BrokerServer id=1] Transition from STARTING to STARTED (kafka.server.BrokerServer)

From a second terminal, confirm the broker answers a metadata request. The cleanest probe is kafka-broker-api-versions.sh because it forces a full handshake on the configured listener and prints every API the broker supports.

bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092

# localhost:9092 (id: 1 rack: null) -> (
#   Produce(0): 0 to 12 [usable: 12],
#   Fetch(1): 0 to 17 [usable: 17],
#   ListOffsets(2): 0 to 9 [usable: 9],
#   Metadata(3): 0 to 13 [usable: 13],
#   ...
# )

If the command hangs or returns “Connection refused,” the broker is not bound to the port you expect. Check logs/server.log for a “Listeners port already in use” line, then either change the port in tutorial.properties or stop the conflicting process. Port 9092 is the canonical Kafka default and is also the convention every client library expects.

Step 5: Create Topics, Partitions, and Replication Factor

Kafka stores every event in a topic. Each topic is split into partitions (the unit of parallelism) and each partition is replicated across replicas for durability. On a single-node cluster we can only have a replication factor of 1, so plan to revisit those numbers when you move to a real cluster. The recommendation is replication factor 3 in production with min.insync.replicas=2, which tolerates one broker loss without producer data loss.

# Create a topic with 3 partitions
bin/kafka-topics.sh --create \
  --bootstrap-server localhost:9092 \
  --topic orders \
  --partitions 3 \
  --replication-factor 1

# Created topic orders.

# List topics
bin/kafka-topics.sh --list --bootstrap-server localhost:9092
# orders

# Describe a topic
bin/kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic orders
# Topic: orders TopicId: ...  PartitionCount: 3  ReplicationFactor: 1
#  Topic: orders Partition: 0 Leader: 1 Replicas: 1 Isr: 1
#  Topic: orders Partition: 1 Leader: 1 Replicas: 1 Isr: 1
#  Topic: orders Partition: 2 Leader: 1 Replicas: 1 Isr: 1

How to Pick a Partition Count

Pick partitions based on target throughput, not on a fixed multiplier. A useful rule from the Confluent reference architectures is to size partitions so that a single partition can handle peak per-key traffic on a single consumer, and then scale out by partition count. Start small (3 to 12 for most apps), measure, and add more partitions later. You can grow the partition count of a topic at runtime, but you cannot shrink it; the rebalancing of the keyed messages would break ordering.

Step 6: Produce and Consume from the Command Line

Before writing any client code, verify the round trip with the bundled console tools. kafka-console-producer.sh reads stdin and sends each line as a record. kafka-console-consumer.sh attaches to a topic and prints what arrives. They are the Kafka equivalent of echo and cat.

# Terminal A: producer
bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders
>{"order_id":1001,"sku":"A1","qty":2}
>{"order_id":1002,"sku":"B7","qty":1}
>^D

# Terminal B: consumer
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders --from-beginning
# {"order_id":1001,"sku":"A1","qty":2}
# {"order_id":1002,"sku":"B7","qty":1}

The --from-beginning flag rewinds to offset 0. Without it, the consumer would only print events sent after it joined. That offset behavior is configurable through auto.offset.reset, which can be latest (the default), earliest, or none. In real applications you almost always want latest for new consumer groups and rely on the committed offset for restarts.

Step 7: Build a Java Producer With the Official Client

The Kafka command-line tools are great for smoke tests, but real services use the language clients. The Java client is the reference implementation and the only one that ships with Apache Kafka itself. Create a tiny Maven project that depends on org.apache.kafka:kafka-clients:4.2.0 and writes a few records to the orders topic with idempotent delivery enabled.

// pom.xml dependency
<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka-clients</artifactId>
  <version>4.2.0</version>
</dependency>

// src/main/java/com/example/OrderProducer.java
package com.example;

import org.apache.kafka.clients.producer.*;
import java.util.Properties;

public class OrderProducer {
  public static void main(String[] args) throws Exception {
    Properties props = new Properties();
    props.put("bootstrap.servers", "localhost:9092");
    props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    props.put("enable.idempotence", "true");
    props.put("acks", "all");
    props.put("compression.type", "zstd");

    try (Producer<String, String> producer = new KafkaProducer<>(props)) {
      for (int i = 0; i < 10; i++) {
        ProducerRecord<String, String> record =
          new ProducerRecord<>("orders", "user-" + i, "{\"qty\":" + i + "}");
        producer.send(record, (metadata, ex) -> {
          if (ex != null) ex.printStackTrace();
          else System.out.printf("offset=%d, partition=%d%n", metadata.offset(), metadata.partition());
        });
      }
      producer.flush();
    }
  }
}

Three configuration choices in that snippet matter. enable.idempotence=true makes the producer attach a sequence number to every record so the broker can deduplicate retries; this is on by default in 4.x but worth setting explicitly. acks=all forces the leader to wait for the in-sync replicas before acknowledging. compression.type=zstd usually gives 30 to 50 percent better ratios than gzip on JSON payloads with very low CPU overhead. Build with mvn package and run with java -jar target/order-producer.jar.

Step 8: Build a Python Consumer With confluent-kafka

Python is the most common second language for Kafka clients, especially in data science teams. The recommended library is confluent-kafka, a thin wrapper around the high-performance librdkafka C client. It is faster than pure-Python alternatives like kafka-python and supports the latest broker features within weeks of each Kafka release.

python3 -m venv .venv
source .venv/bin/activate
pip install confluent-kafka

# orders_consumer.py
from confluent_kafka import Consumer, KafkaError

conf = {
    "bootstrap.servers": "localhost:9092",
    "group.id": "order-fulfillment",
    "auto.offset.reset": "earliest",
    "enable.auto.commit": False,
}

consumer = Consumer(conf)
consumer.subscribe(["orders"])

try:
    while True:
        msg = consumer.poll(1.0)
        if msg is None:
            continue
        if msg.error():
            if msg.error().code() == KafkaError._PARTITION_EOF:
                continue
            raise Exception(msg.error())
        print(f"offset={msg.offset()} key={msg.key()} value={msg.value()}")
        consumer.commit(asynchronous=False)
finally:
    consumer.close()

Two patterns in this consumer are worth highlighting. The group.id identifies a consumer group, the unit of parallelism on the read side; partitions of the orders topic will be split across all members of the group. We turn off auto-commit and call consumer.commit() after we have processed each record, which gives at-least-once delivery semantics. Auto-commit is convenient but combined with synchronous processing it leads to message loss when a worker dies between commit and processing.

Step 9: Use Share Groups (Queues for Kafka, KIP-932)

One of the headline features of Kafka 4.2 is the general availability of Queues for Kafka, internally called Share Groups. A traditional Kafka consumer group assigns each partition to exactly one consumer; if you have ten partitions you can scale to ten readers and no further. A share group lets multiple consumers cooperatively read from the same partition, with broker-side acknowledgement of individual records. That is exactly the pattern RabbitMQ and ActiveMQ users have asked for since the beginning.

In the 4.2 release the share-group protocol added the RENEW acknowledgement type for long-running handlers, adaptive batching, and lag metrics that match the consumer-group dashboards Confluent and Grafana already ship. Read the original proposal in KIP-932 for the full state machine, including the difference between ACCEPT, RELEASE, and REJECT.

# Enable share groups in your tutorial.properties
echo "group.coordinator.rebalance.protocols=classic,consumer,share" \
  >> config/kraft/tutorial.properties
echo "share.coordinator.state.topic.replication.factor=1" \
  >> config/kraft/tutorial.properties

# Restart the broker, then create a queue-style consumer
bin/kafka-console-share-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic orders \
  --group payment-workers

Run two instances of kafka-console-share-consumer.sh with the same --group and produce 100 records. Both consumers will receive a roughly even split of the records even though the topic only has three partitions. That is the queue semantics that previously required a second system.

Step 10: Stream Processing With Kafka Streams and Dead Letter Queues

Kafka Streams is a Java library for stateful transformations that runs inside your application, not on a separate cluster. Think of it as the Kafka-native answer to Apache Flink for the common case of map-filter-aggregate-join over event streams. The 4.2 release added Dead Letter Queue support through KIP-1034, so deserialization errors and processor exceptions can be routed to a side topic for inspection rather than killing the whole topology.

// build.gradle dependency
implementation 'org.apache.kafka:kafka-streams:4.2.0'

// OrderEnricher.java
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.streams.*;
import org.apache.kafka.streams.kstream.*;
import java.util.Properties;

public class OrderEnricher {
  public static void main(String[] args) {
    Properties props = new Properties();
    props.put(StreamsConfig.APPLICATION_ID_CONFIG, "order-enricher-v1");
    props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
    props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());
    // KIP-1034: route bad records here instead of crashing
    props.put("default.deserialization.exception.handler",
      "org.apache.kafka.streams.errors.DeadLetterQueueExceptionHandler");
    props.put("errors.deadletterqueue.topic.name", "orders-dlq");

    StreamsBuilder builder = new StreamsBuilder();
    KStream<String, String> orders = builder.stream("orders");
    orders.filter((k, v) -> v != null && v.contains("\"qty\""))
          .mapValues(v -> v.replace("}", ",\"enriched\":true}"))
          .to("orders-enriched");

    new KafkaStreams(builder.build(), props).start();
  }
}

Run the producer from Step 7 again and you will see records flow into orders-enriched. Push a malformed JSON record by hand from the console producer and watch it land in orders-dlq instead of crashing the topology. The new Streams Rebalance Protocol from KIP-1071 also kicks in here: when you start a second instance of the same application.id, the broker assigns the new tasks centrally instead of the clients negotiating, and the rebalance time on a 12-task topology drops from seconds to under 200 milliseconds in our local tests.

Step 11: Tiered Storage and Long-Term Retention

Tiered Storage (KIP-405) graduated to GA in Kafka 3.8 and has been hardening through the 4.x line. It splits each partition into a “local” tier on the broker disk for the recent hot data and a “remote” tier in object storage (S3, GCS, Azure Blob) for older segments. The result is the same wire protocol but with effectively unbounded retention at object-storage prices, around $0.023 per GB-month on S3 standard versus $0.10+ for EBS gp3.

# Enable tiered storage on the broker (S3 example, requires plugin JAR on classpath)
remote.log.storage.system.enable=true
remote.log.metadata.manager.class.name=org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager
remote.log.storage.manager.class.name=com.example.S3RemoteStorageManager
remote.log.storage.manager.impl.prefix=s3.

# Mark a topic as tier-eligible
bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name orders \
  --add-config remote.storage.enable=true,local.retention.ms=3600000,retention.ms=2592000000

The settings above keep one hour of data on the broker’s local disk and offload everything older to remote storage, with a hard 30-day retention window. AWS MSK and Confluent Cloud both expose this as a checkbox on the topic config; running it on your own cluster requires a tiered storage plugin from your object store vendor or the open-source Aiven plugin.

Step 12: Monitor With JMX, Prometheus, and the Admin API

Kafka exposes hundreds of JMX metrics. The most important for this Apache Kafka tutorial are UnderReplicatedPartitions (should be 0), RequestHandlerAvgIdlePercent (should be above 0.3), ConsumerLagSum per group (should be bounded), and the new kafka.controller.idle-ratio introduced in 4.2 through KIP-1227. The simplest way to scrape JMX into Prometheus is the official JMX exporter agent.

# Download the agent and config
curl -L -o /tmp/jmx_prometheus_javaagent.jar \
  https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/1.0.1/jmx_prometheus_javaagent-1.0.1.jar

# Start Kafka with the agent attached
KAFKA_OPTS="-javaagent:/tmp/jmx_prometheus_javaagent.jar=7071:config/jmx_exporter.yml" \
  bin/kafka-server-start.sh config/kraft/tutorial.properties

# Scrape it
curl -s http://localhost:7071/metrics | grep kafka_server_brokertopicmetrics_messagesin_total

Pair the JMX scraper with Grafana and the official Confluent Kafka dashboard ID 11962. For higher-level health, the AdminClient API exposes the cluster topology, in-sync replicas, consumer-group lag, and now (in 4.2) the rack ID of every group member, which is invaluable in multi-AZ deployments.

Step 13: Deploy to Kubernetes With Strimzi or to Confluent Cloud

Once your local cluster works, the easiest production path is the Strimzi Cluster Operator on Kubernetes or a managed service like AWS MSK or Confluent Cloud. Strimzi 0.51 adds support for Kafka 4.2.0; install it with a single Helm chart and describe your cluster declaratively.

# Strimzi 0.51 install
helm repo add strimzi https://strimzi.io/charts/
helm install strimzi strimzi/strimzi-kafka-operator --namespace kafka --create-namespace

# kafka-cluster.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: tutorial
  namespace: kafka
spec:
  kafka:
    version: 4.2.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
    storage:
      type: persistent-claim
      size: 100Gi
  entityOperator:
    topicOperator: {}
    userOperator: {}

kubectl apply -f kafka-cluster.yaml
kubectl -n kafka wait kafka/tutorial --for=condition=Ready --timeout=300s

For AWS, the recommended pattern is MSK Serverless behind PrivateLink, which removes the need to size brokers and scales partitions automatically. The trade-off is cost: MSK Serverless bills per partition-hour and per GB ingested, which is competitive at moderate scale and expensive at very high throughput. For long-running steady workloads, MSK Provisioned with reserved instances is cheaper. Confluent Cloud is the most feature-complete managed offering: it ships Schema Registry, ksqlDB, Stream Governance, and audit logs in one package.

Common Pitfalls and How to Avoid Them

Six failure modes catch new Kafka operators in the first month. Each one has a clean fix once you know what to look for.

• Pitfall 1: Default replication factor of 1 in production. A single broker failure becomes data loss. Always set default.replication.factor=3 and min.insync.replicas=2 on the broker, and require acks=all on the producer.
• Pitfall 2: Too many partitions. Every partition consumes memory in the broker and a file handle on disk. Clusters with hundreds of thousands of partitions hit OOM under load. Stick to under 4,000 partitions per broker until you have profiled.
• Pitfall 3: Long consumer-group rebalance storms. Old protocols pause every consumer when one joins. Switch to KIP-848 cooperative rebalancing by setting group.protocol=consumer on clients running against a 4.x broker.
• Pitfall 4: Producer message ordering broken by retries. If enable.idempotence=false and max.in.flight.requests.per.connection > 1, retries can reorder messages. Either keep idempotence on (the 4.x default) or set in-flight to 1.
• Pitfall 5: Open file limit too low. Brokers under load easily open 100,000+ files. Set nofile=1000000 in /etc/security/limits.conf and verify with ulimit -n before starting the broker.
• Pitfall 6: Mixing KRaft and ZooKeeper configs. In 4.x, any zookeeper.connect entry in your config will be silently ignored. Read every property file before pasting from old tutorials.

Troubleshooting Apache Kafka 4.2: 8 Real Problems

• “InconsistentClusterIdException” on startup. Your log.dirs contains a meta.properties from another cluster. Stop the broker, delete log.dirs, re-run kafka-storage.sh format.
• “ListenerNotFoundException” from clients. The advertised listener does not match the listener clients are connecting to. Set both listeners and advertised.listeners in tutorial.properties.
• Consumer never reads any record. Either the group.id already committed past the data (set auto.offset.reset=earliest and use a fresh group), or the topic name is misspelled.
• “NotEnoughReplicasException” on produce. The number of in-sync replicas is below min.insync.replicas. Either bring back the offline broker or temporarily lower min.insync.replicas while you investigate.
• Disk filling up overnight. Default retention is 7 days; with 50 MB/sec ingest you accumulate 30 TB. Lower retention.ms per topic or enable Tiered Storage.
• JVM running out of memory. The broker default heap is 1 GB. For more than light load, set KAFKA_HEAP_OPTS="-Xms6g -Xmx6g" before starting.
• Slow consumer that never catches up. Check fetch.max.bytes and max.poll.records. The defaults are conservative; raising them can double or triple per-consumer throughput.
• “OFFSET_OUT_OF_RANGE” on resume. Your committed offset points to a segment already deleted by retention. Reset with kafka-consumer-groups.sh --reset-offsets --to-earliest.

Apache Kafka vs RabbitMQ vs Pulsar: When to Pick What

Kafka is not the right hammer for every nail. Where it shines: long-retention event logs, very high throughput (hundreds of MB/sec per broker), stream processing with Kafka Streams or Flink, and replay use cases. Where other systems can be a better fit: classic work queues with per-message acknowledgement (RabbitMQ historically; Kafka Share Groups now compete here), multi-tenant geo-replicated topics with per-topic isolation (Pulsar’s strong suit), and cron-style scheduled delivery (NATS or Temporal). Our deeper benchmark in the RabbitMQ vs Kafka 2026 head-to-head measured 1M msg/sec on Kafka against 50K on RabbitMQ for a fan-out workload, but the gap narrows considerably when message size is small and latency targets are aggressive.

Use case	Best fit	Why
Event log with replay	Apache Kafka	Native log abstraction, Tiered Storage, days/weeks retention
Stream processing	Apache Kafka + Streams or Flink	Topic-as-table model, exactly-once semantics
Classic work queue	Kafka Share Groups or RabbitMQ	Per-record ack with KIP-932 closes the gap
Multi-tenant SaaS	Apache Pulsar	Per-topic backlog isolation
RPC / request-reply	gRPC, NATS	Kafka adds latency; see our gRPC vs REST 2026
Database CDC pipeline	Apache Kafka + Debezium	Compact topics, Schema Registry, mature connectors

Advanced Tips for Production Apache Kafka

Once the basics work, four production patterns will save you from the most common late-night incidents.

• Schema-first topics. Use Avro or Protobuf with the Confluent Schema Registry, or the open-source Apicurio Registry. Schemaless JSON is the leading cause of breaking-change incidents in Kafka pipelines after 12 months.
• Rack-aware replica placement. Set broker.rack to your AZ. Kafka 4.2 also exposes group-member rack IDs through the Admin API (KIP-1227), which makes follower fetching across AZs both correct and cheap.
• Quotas everywhere. Producer/consumer/request quotas via kafka-configs.sh stop a noisy neighbor from consuming the entire broker. Set them per client ID and per user from day one, not after the first incident.
• Use the new consumer protocol. Set group.protocol=consumer on every client to opt into the KIP-848 server-side rebalance. Cooperative rebalancing alone routinely cuts rebalance time on 50-instance services from 30 seconds to under 1 second.

Apache Kafka Tutorial: The Complete Working Project

Putting every step together, the working project at the end of this tutorial is a four-process system on a single laptop: a KRaft Kafka 4.2 broker, a Java producer that writes idempotent JSON orders, a Python consumer that prints them with manual offset commits, and a Kafka Streams app that enriches each order and writes the result to orders-enriched while routing bad records to orders-dlq. Drop two share-group consumers in front of it and you also have a queue-style processing layer that ships with no extra dependency.

The same code unchanged moves to a 3-broker Strimzi cluster on Kubernetes, to AWS MSK, or to Confluent Cloud by editing exactly one property: bootstrap.servers. That is the single biggest reason the Kafka API has become the lingua franca of streaming: the application code does not change between dev, staging, and prod, even when the underlying cluster goes from three brokers to three hundred.

Frequently Asked Questions

Is Apache Kafka free?

Yes. Apache Kafka is licensed under Apache 2.0 and the binaries on kafka.apache.org/downloads are free for any use, including commercial. Confluent and AWS sell managed services on top, with their own client libraries and additional tools, but the core broker and Kafka Streams library are free software you can run on your own infrastructure.

Do I still need ZooKeeper for Kafka 4.2?

No. ZooKeeper was deprecated in Kafka 3.5 and removed entirely in Kafka 4.0. KRaft, the built-in Raft-based metadata service, is the only supported mode in 4.x. If you are migrating from 2.x or 3.x, the Kafka 3.9 release ships migration tooling that walks the cluster off ZooKeeper without downtime; do that before upgrading to 4.2.

How is Kafka different from RabbitMQ?

Kafka is a partitioned log; RabbitMQ is a routing-based queue. Kafka stores every event for the configured retention (days, weeks, or forever via Tiered Storage) so consumers can replay history. RabbitMQ deletes a message once it is acknowledged. With Kafka 4.2 Share Groups you can now also do queue-style consumption on Kafka, which closes the historical use-case gap.

How many partitions should each topic have?

Start small. For most applications, 3 to 12 partitions per topic on a 3-broker cluster is enough. Scale by adding partitions as throughput grows; you can grow but not shrink. Aim for fewer than 4,000 partitions per broker overall, including replicas, to stay within the file handle and memory budgets that the JVM is comfortable with.

What is KIP-848 and why does it matter?

KIP-848 is the next-generation consumer rebalance protocol that moves rebalance decisions from the clients to the broker. Practically, it cuts rebalance time from seconds to milliseconds on big consumer groups, eliminates the “stop the world” pause, and is fully compatible with the existing consumer API. Opt in with group.protocol=consumer on a 4.x broker.

Can Apache Kafka run on Windows?

The shell scripts ship a .bat equivalent for Windows, and the broker will start. But Apache and most cloud providers do not officially support Windows for production Kafka, and the file-handle and page-cache behavior is meaningfully different on NTFS. The supported path on Windows is WSL2 or a Linux container.

What is the difference between Kafka Connect and Kafka Streams?

Kafka Connect is a framework for moving data between Kafka and external systems (databases, S3, Elasticsearch) using configurable plugins. Kafka Streams is a Java library for transforming data within Kafka. Connect runs as its own process; Streams runs inside your application. They are complementary and both ship with Apache Kafka.

How fast can a single Kafka broker go?

On commodity NVMe-equipped hardware a single broker comfortably sustains hundreds of MB per second of ingest with sub-10ms median producer latency. Real numbers vary widely with replication factor, batch size, compression, and TLS overhead. The Apache documentation and Confluent’s reference benchmarks are the trustworthy starting point; ignore vendor benchmarks that do not publish their producer config.

Is Kafka good for microservices?

Yes for asynchronous communication and event sourcing. No for synchronous request/response, where gRPC is a better fit. Many teams pair both: gRPC for the user-facing API path, Kafka for everything that fans out, retains, or feeds analytics.

What does Tiered Storage cost compared to local disks?

Object storage like S3 Standard runs about $0.023 per GB-month versus $0.10+ per GB-month for EBS gp3. For workloads that retain weeks of data, Tiered Storage commonly cuts the storage bill by 70 to 80 percent while keeping the hot tier on the broker for fast reads.

What expert quote should I trust on Kafka 4.x?

“Removing ZooKeeper isn’t just an operational cleanup; it removes an entire failure domain and dramatically simplifies how teams reason about cluster state,” wrote Jun Rao, Kafka co-creator and Confluent co-founder, in the announcement of the 4.0 release. The 4.2 release continues that arc: every new feature in 4.2 ships only on KRaft, and the migration window for ZooKeeper-based clusters formally closed in early 2026.

Related Coverage

• RabbitMQ vs Kafka 2026: 1M vs 50K msg/sec and a 16x Gap [Tested]
• gRPC vs REST 2026: 77% Faster, 10x Smaller Payloads
• Spring Boot Tutorial: Build a REST API in 13 Steps [2026]
• How to Master Redis with Python: 12-Step Tutorial
• How to Build a Task Queue with Celery Python and Redis in 13 Steps
• PostgreSQL vs MySQL 2026: 3.7x JSON Speed Gap
• SQL vs NoSQL 2026: 48% vs 25% Use and 5x Throughput

If this Apache Kafka tutorial helped you go from zero to a working KRaft cluster, the next step is to instrument the cluster (Step 12), pick a managed deployment target (Step 13), and read every apache/kafka release note from 4.0 forward so you understand what shipped, what got removed, and what is on the way for 4.3.