Apache Kafka is a popular distributed event streaming platform. In modern Spring Boot applications, integrating Kafka with observability tools like Micrometer enables you to gain visibility into producer and consumer behavior. Let us delve into understanding how Spring Kafka Micrometer integration helps monitor and trace Kafka-based applications effectively.
1. Introduction
Apache Kafka is a high-throughput, distributed messaging system that enables real-time processing of streaming data. It allows applications to publish, subscribe to, store, and process streams of records in a fault-tolerant manner. Kafka is often used in microservices, event sourcing, and real-time analytics architectures. Key features of Apache Kafka include horizontal scalability, high durability, support for high-throughput message processing, strong ordering guarantees within partitions, and fault tolerance via replication. It also offers decoupling between producers and consumers, stream retention for replayability, and integration with stream processing frameworks like Kafka Streams and Apache Flink.
With the integration of Micrometer into Spring Kafka, developers can seamlessly monitor and trace message flows, latency, and error rates across Kafka producers and consumers. This combination empowers observability in distributed systems by emitting metrics compatible with popular monitoring systems like Prometheus and Grafana.
1.1 Kafka Docker Setup
To quickly set up Kafka for development or testing, you can use Docker Compose. This setup provisions both Zookeeper and Kafka using the official Confluent Platform Docker images. Zookeeper is essential for Kafka as it manages cluster metadata and broker coordination.
# docker-compose.yml
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:7.5.0
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ZOOKEEPER_SYNC_LIMIT: 2
kafka:
image: confluentinc/cp-kafka:7.5.0
depends_on:
- zookeeper
ports:
- "9092:9092"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
This docker-compose.yml file sets up a basic Kafka and Zookeeper environment using the Confluent Platform Docker images. It defines two services: zookeeper and kafka. The Zookeeper service uses the confluentinc/cp-zookeeper:7.5.0 image and is configured with default settings such as the client port 2181 and basic synchronization parameters. The Kafka service uses the confluentinc/cp-kafka:7.5.0 image and depends on Zookeeper to start. It exposes Kafka on port 9092 and is configured with environment variables to specify its broker ID, Zookeeper connection, advertised listener address, and security protocol mapping. It also enables automatic topic creation and sets the replication factor for internal topics like offsets to 1, which is suitable for development or local testing environments.
1.1.1 Running the Container
To bring up the services, run the following command:
docker-compose up -d
After startup, verify Kafka by listing topics. Replace <kafka_container_id_or_name> with the actual Kafka container name.
docker exec -it <kafka_container_id_or_name> kafka-topics --bootstrap-server localhost:9092 --list
You can also create a topic and test Kafka locally using CLI tools.
# Create a topic docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --create --topic demo-topic --partitions 1 --replication-factor 1
2. Code Example
2.1 Project Setup
The project is a Spring Boot Kafka application instrumented with Micrometer and Spring Boot Actuator for observability. The following Maven dependencies are required in pom.xml:
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-core</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
</dependency>
</dependencies>
These dependencies enable Kafka producer/consumer capabilities, expose metrics via Actuator, and export them to Prometheus.
2.2 application.yml
The application.yml configures Kafka connectivity and enables metrics exposure for monitoring. Make sure Kafka is running at localhost:9092 before starting the Spring Boot app.
spring:
kafka:
bootstrap-servers: localhost:9092
consumer:
group-id: demo-group
auto-offset-reset: earliest
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
producer:
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.apache.kafka.common.serialization.StringSerializer
management:
endpoints:
web:
exposure:
include: '*'
endpoint:
prometheus:
enabled: true
metrics:
export:
prometheus:
enabled: true
observations:
key-values:
application: spring-kafka-demo
2.2.1 Configuration file Explanation
This application.yml file configures Kafka and Micrometer settings for a Spring Boot application. Under the spring.kafka section, it defines Kafka connection details with the bootstrap-servers set to localhost:9092. The consumer configuration specifies a group-id of demo-group, sets auto-offset-reset to earliest for reading messages from the beginning if no offset is present, and uses the standard string deserializers for keys and values. Similarly, the producer section defines string serializers for outgoing Kafka messages. The management section enables Spring Boot Actuator’s web endpoints, including the Prometheus metrics endpoint, by setting exposure.include to * and explicitly enabling prometheus. Additionally, it configures Micrometer observations with a custom application tag set to spring-kafka-demo, which helps identify and group metrics under a consistent label when visualizing data in monitoring tools like Prometheus and Grafana.
2.3 Producer Service with Micrometer Observation
The producer service uses Micrometer’s Observation API to instrument Kafka message publishing. This allows visibility into the latency and frequency of message production, enabling Prometheus or other observability tools to collect and analyze metrics.
@Service
public class KafkaProducer {
private final KafkaTemplate<String, String> kafkaTemplate;
private final ObservationRegistry observationRegistry;
public KafkaProducer(KafkaTemplate<String, String> kafkaTemplate, ObservationRegistry observationRegistry) {
this.kafkaTemplate = kafkaTemplate;
this.observationRegistry = observationRegistry;
}
public void sendMessage(String topic, String message) {
Observation.createNotStarted("kafka.send", observationRegistry)
.lowCardinalityKeyValue("topic", topic)
.observe(() -> {
kafkaTemplate.send(topic, message);
System.out.println("Produced message: " + message);
return null;
});
}
}
2.3.1 Code Explanation
The KafkaProducer class is a Spring @Service responsible for sending messages to a Kafka topic while capturing observability data using Micrometer’s Observation API. It has two dependencies injected via constructor: KafkaTemplate<String, String> for publishing messages and ObservationRegistry for recording metrics. The sendMessage() method begins by creating a custom observation named kafka.send, associated with the provided registry, and adds a low-cardinality tag to indicate the Kafka topic. The observe() method wraps the actual message-sending logic to measure execution time and attach metadata. Inside this block, the message is published using kafkaTemplate.send(), and a log is printed to confirm the message was produced. This observability setup enables the collection of custom metrics such as message send duration, frequency, and topic-level details, which can then be exposed via the Spring Boot Actuator’s Prometheus endpoint and visualized using tools like Grafana.
2.4 Consumer Listener
The consumer listens on the topic demo-topic and logs the received message. This basic implementation can be extended to include custom observation events for message consumption.
@Component
public class KafkaConsumer {
@KafkaListener(topics = "demo-topic", groupId = "demo-group")
public void listen(String message) {
System.out.println("Consumed message: " + message);
}
}
2.4.1 Code Explanation
The KafkaConsumer class is annotated with @Component, making it a Spring-managed bean responsible for consuming messages from a Kafka topic. It contains a single method, listen(), annotated with @KafkaListener, which configures it to listen to the topic demo-topic as part of the consumer group demo-group. When a message is received on the specified topic, the method is automatically triggered, and the message payload is passed as a parameter. The method then logs the consumed message using System.out.println(). Spring Kafka handles the underlying consumer creation and message polling, allowing developers to focus on business logic while ensuring reliable and scalable message consumption.
2.5 REST Controller to Trigger Kafka Send
This controller exposes a POST API that allows clients to send a message to Kafka via HTTP. Internally, it invokes the producer service method.
@RestController
@RequestMapping("/api/kafka")
public class KafkaController {
private final KafkaProducer kafkaProducer;
public KafkaController(KafkaProducer kafkaProducer) {
this.kafkaProducer = kafkaProducer;
}
@PostMapping("/send")
public ResponseEntity<String> send(@RequestParam String message) {
kafkaProducer.sendMessage("demo-topic", message);
return ResponseEntity.ok("Message sent!");
}
}
2.5.1 Code Explanation
The KafkaController class is annotated with @RestController and handles HTTP requests related to Kafka messaging under the base path /api/kafka. It depends on the KafkaProducer service, which is injected via constructor. The controller exposes a POST endpoint at /send, which accepts a request parameter named message. When this endpoint is called, the send() method invokes the sendMessage() method of the producer to publish the message to the Kafka topic demo-topic. Upon successful publishing, it returns a 200 OK response with the text "Message sent!". This setup allows users to trigger Kafka message production over HTTP using tools like Postman or curl.
2.6 Running the Example
Start the Spring Boot application using your IDE or by running:
./mvnw spring-boot:run
Once the application is running on localhost:8080, you can test the Kafka message flow using Postman or a simple curl command:
curl -X POST "http://localhost:8080/api/kafka/send?message=HelloKafka"
This sends the message “HelloKafka” to the topic demo-topic via the REST endpoint. The output will appear in the application logs:
Produced message: HelloKafka Consumed message: HelloKafka
The Micrometer Observation added in the producer will automatically record metrics about the message send operation, which are exposed via Prometheus-compatible endpoints.
2.6.1 Micrometer Metrics Endpoint
To view available metrics, visit the actuator metrics endpoint in your browser:
http://localhost:8080/actuator/metrics
Useful Kafka-related metrics include:
/actuator/metrics/spring.kafka.producer– This endpoint exposes Micrometer metrics related to Kafka producers managed by Spring Kafka. It includes details such as the number of records sent, record send rate, error counts, and total send duration. These metrics help in monitoring the producer’s performance, throughput, and reliability. Here is the JSON response that the API will return.{"name":"spring.kafka.producer.record.send.total","description":"Total number of records sent","baseUnit":"messages","measurements":[{"statistic":"COUNT","value":5}],"availableTags":[{"tag":"topic","values":["demo-topic"]}]}/actuator/metrics/spring.kafka.listener– This endpoint provides metrics about Kafka listeners (consumers) in the application. It includes data like the number of records consumed, processing time, and consumer lag. These insights are useful for tracking the efficiency and responsiveness of message consumption. Here is the JSON response that the API will return.{"name":"spring.kafka.listener.records","description":"Number of records received","baseUnit":"messages","measurements":[{"statistic":"COUNT","value":5}],"availableTags":[{"tag":"listener.id","values":["org.springframework.kafka.KafkaListenerEndpointContainer#0"]},{"tag":"topic","values":["demo-topic"]}]}/actuator/metrics/kafka.send(custom observation) – This is a user-defined observation metric created using Micrometer’sObservationAPI in the producer service. It captures timing and tagging information for each Kafka message send operation, such as the topic name and execution duration. It provides fine-grained observability specific to business or operational needs. Here is the JSON response that the API will return.{"name":"kafka.send","description":"Custom observation for Kafka send operation","baseUnit":"milliseconds","measurements":[{"statistic":"TOTAL_TIME","value":112.4},{"statistic":"COUNT","value":5},{"statistic":"MAX","value":32.8}],"availableTags":[{"tag":"topic","values":["demo-topic"]},{"tag":"application","values":["spring-kafka-demo"]}]}
These endpoints display real-time metrics including counts, error rates, and latency histograms. You can also configure a Prometheus server to scrape these metrics from /actuator/prometheus and visualize them using Grafana.
3. Conclusion
Observability is critical in Kafka-based microservices. With Spring Kafka and Micrometer Observation, you can effectively trace, monitor, and analyze Kafka producer and consumer behavior. Add distributed tracing to complete your observability stack.
