Apache Kafka 4.1: The 3 Big Things Developers Need To Know
Kafka 4.1 includes something for all data-streaming enthusiasts. It addresses three areas where developers have been asking for native solutions.
Sep 8th, 2025 10:00am by
Image from Profit_Image on Shutterstock.
Confluent sponsored this post.
Evolution of Queues for Kafka
The 4.1 release moves feature KIP-932 from “early access” to “preview,” meaning there’s been a huge effort to make Queues for Kafka more stable, while also adding new functionality. To quickly recap, KIP-932 introduces the concept of cooperative consumption. This allows for multiple consumers to process messages from the same topic partitions, rather than the “exclusive assignment” we see with traditional Kafka Consumers. When we layer in per-message acknowledgement and delivery counts to facilitate retries, this enables Kafka to handle a set of use cases that are typically built around the concept of a queue. Think of scenarios where a message represents an individual unit of work or a task, and a pool of applications could independently process each message — acknowledging a message as processing completes. A queue is a better fit for this scenario than a partitioned topic in a consumer group. KIP-932 introduces a new kind of group called a share group that provides this new behavior. The newKafkaShareConsumer class includes a configuration property that allows applications fine-grained control over the life cycle and acknowledgement of each message. By setting the property share.acknowledgment.mode to explicit, the client can now use the values of the AcknowledgmentType enumeration to specify how a message moves through the state machine of the share partition.
When a message is processed successfully, use the acknowledge method of the KafkaShareConsumer with the ACCEPT acknowledgement type to move this message to the Acknowledged stage of the life cycle. The real control here comes from exception cases. The developer can determine where to RELEASE or REJECT the ConsumerRecord based on the type of exception that occurred during processing. For errors that can be retried, use RELEASE to (potentially) send the message back to the Available state. If the error isn’t recoverable or the message is erroneous, use REJECT to send the message to the Archived state and move on with processing others.
Here’s how this looks in practice:
- First, we configure and instantiate a
KafkaShareConsumer. - That consumer instance will
subscribeto one or more topics. - Calling poll will return a batch of
ConsumerRecords. - Iterating over those records, we attempt to process each message in the
doProcessing()method. - If that processing succeeds, we
acknowledge()that record with theAcknowledgementType.ACCEPT. - If we catch an Exception, we
acknowledge()that record with theAcknowledgementType.RELEASE. - For the control freaks among us, explicitly calling the
KafkaShareConsumer.commitSync()method will commit the message-level acknowledgements from this batch. Otherwise, the acknowledgements are automatically committed.
public static void main(String[] args) {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "my-share-group");
props.put("share.acknowledgement.mode", "explicit");
KafkaShareConsumer<String, String> consumer = new KafkaShareConsumer<>(
props, new StringDeserializer(), new StringDeserializer());
// subscribe to a topic, joining the share group
consumer.subscribe(List.of("foo"));
while (true) {
// Fetch a batch of records acquired for this consumer...
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
records.forEach(record -> {
try {
doProcessing(record);
// Acknowledge this event was handled without error.
consumer.acknowledge(record, AcknowledgeType.ACCEPT);
} catch (Exception e) {
logger.error("Consumer: Error handling event: {}", e.getMessage(), e);
// NOTE: Choosing to RELEASE on any exception here.
consumer.acknowledge(record, AcknowledgeType.RELEASE);
}
});
// OPTIONALLY: Commit the Acknowledgements of this batch of records.
//consumer.commitSync();
}
}
Native JWT-Bearer Support Simplifies Kafka Authentication
Until now, securing your Kafka client applications often involved managing static credentials using theclient_credentials grant type. An alternative was building and maintaining some custom integrations with Kafka’s pluggable authentication interface, perhaps with an identity provider like Okta, AWS IAM, Google Cloud IAM or Auth0. These resulted in long-lived, static credentials or required organizations to build and maintain a complex, often unsupported bespoke solution to integrate Kafka clients with their modern identity provider.
With KIP-1139, Kafka clients now support the jwt-bearer token grant type, enabling clients to authenticate securely using industry-standard JSON Web Tokens (JWTs) from their existing OAuth 2.o and OpenID Connect (OIDC) identity providers. This eliminates the need for static credentials. Enterprises with strong security requirements can use this native support of jwt-bearer for all Kafka client applications: producers, consumers and Kafka connectors.
Consult your cloud provider’s documentation on how to obtain the connection information needed by your clients and brokers. To learn how to configure your identity provider to leverage the functionality of jwt-bearer token grants, see this Confluent documentation.
Kafka Streams Rebalance Protocol
The early access release of KIP-1071 begins to apply the lessons from KIP-848 (The Next Generation of the Consumer Group Rebalance Protocol) to Kafka Streams applications. As a refresher, with KIP-848, Kafka brokers now coordinate rebalancing consumer groups and eliminate “stop the world” scenarios in event processing. Applying this to Kafka streams means the brokers can also coordinate task assignments and rebalancing decisions for Kafka Streams topologies. This provides a “streams-optimized” coordination separate from the traditional consumer group protocol. KIP-1071 introduces a new streams group protocol,group.protocol=streams, that helps differentiate between different types of consumers in your event streaming ecosystem. There are new operations in the Admin API to describe these streams groups, as well as new command line interface (CLI) scripts.
As for monitoring, new broker-side metrics also separate these protocols — be it a traditional consumer group, a share group or a streams application. Increased task-level visibility in your Kafka Streams application will help with debugging and troubleshooting scenarios.
Since this is for an early access feature, it’s strictly for testing purposes and not yet ready for production uses. Kafka Streams developers are encouraged to experiment and provide feedback that will help shape the future of this functionality.
Additional Features in Apache Kafka 4.1
There are many more useful enhancements in the Apache Kafka 4.1 release, including:- KIP-848: Consumer Group Protocol, covering broker and client post-GA follow-ups for broker-driven rebalancing. This provides faster and more stable consumer group rebalances.
- KIP-890: Transactions Server-Side Defense, which improves transaction behavior and correctness.
- KIP-853: KRaft membership controls, specifically upgrading from static to dynamic voters.
- KIP-1050: Introduced enhanced and unified strategy for error handling for transactions.
- KIP-1111: Enforces explicit naming for Kafka Streams Internal Topics.
- KIP-1109: Introduces unified metric names for consumers and producers.
Confluent, founded by the original creators of Apache Kafka, pioneered a complete data streaming platform that streams, connects, processes, and governs data as it flows throughout a business. With Confluent, any organization can modernize their business and run it in real-time.
Learn More
The latest from Confluent
YOUTUBE.COM/THENEWSTACK
Tech moves fast, don't miss an episode. Subscribe to our YouTube
channel to stream all our podcasts, interviews, demos, and more.
