In modern distributed systems, event-driven architectures (EDA) have become a cornerstone for building scalable and resilient applications. Kafka is often the backbone of such architectures, serving as a durable, high-throughput event streaming platform. However, integrating Kafka into complex workflows frequently requires sophisticated routing, transformation, and mediation logic. This is where Apache Camel shines.
Apache Camel is an open-source integration framework that provides a wide range of components (over 300) to connect disparate systems. With Camel’s powerful routing DSLs (Java, XML, YAML), you can declaratively define routes to consume, process, and produce Kafka messages without boilerplate code.
In this article, we’ll explore how to build Camel routes to integrate Kafka in an event-driven system, transform messages, and connect to other systems such as databases or REST APIs.
1. Why Use Apache Camel with Kafka?
Kafka excels at delivering and storing streams of events, but Camel complements it by providing:
- Routing: Declarative routes to define how messages flow through systems.
- Transformation: Convert message formats (e.g., JSON to Avro, CSV to JSON).
- Mediation: Enrich, filter, split, aggregate, and handle errors in streams.
- Connectivity: Integrate Kafka with hundreds of systems (databases, file systems, cloud services).
This combination allows you to decouple producers and consumers, implement complex processing logic, and centralize integration flows.
2. Getting Started: Setting up Camel Kafka Component
Let’s walk through a concrete example: consuming messages from a Kafka topic, transforming them, and forwarding them to another topic.
First, include Camel and Kafka dependencies in your pom.xml:
<dependencies>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-core</artifactId>
<version>4.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-kafka</artifactId>
<version>4.3.0</version>
</dependency>
<!-- Optionally add camel-jackson for JSON processing -->
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-jackson</artifactId>
<version>4.3.0</version>
</dependency>
</dependencies>
3. Example Route: Consuming, Transforming, and Producing Kafka Messages
Below is a simple Camel Java DSL route that:
- Consumes JSON events from the
orderstopic. - Transforms them into a simplified format.
- Produces them to a
processed-orderstopic.
from("kafka:orders?brokers=localhost:9092")
.routeId("kafka-transform-route")
.unmarshal().json()
.process(exchange -> {
Map<String, Object> body = exchange.getIn().getBody(Map.class);
Map<String, Object> transformed = new HashMap<>();
transformed.put("orderId", body.get("id"));
transformed.put("customer", body.get("customer"));
transformed.put("total", body.get("amount"));
exchange.getIn().setBody(transformed);
})
.marshal().json()
.to("kafka:processed-orders?brokers=localhost:9092");
How it works:
from("kafka:orders"): Consumes messages from Kafka..unmarshal().json(): Converts the JSON string into a JavaMap..process(): Extracts and restructures fields..marshal().json(): Converts the Java object back to JSON..to("kafka:processed-orders"): Sends it to another Kafka topic.
4. Advanced Processing: Filtering and Enrichment
Camel allows you to build more advanced pipelines. For example, you might want to filter events with a total > 1000 and enrich them by calling a REST service.
from("kafka:orders?brokers=localhost:9092")
.routeId("kafka-filter-enrich")
.unmarshal().json()
.filter(simple("${body[amount]} > 1000"))
.enrich("https://api.example.com/customer/${body[customerId]}")
.marshal().json()
.to("kafka:high-value-orders?brokers=localhost:9092");
In this example:
.filter(): Filters events based on amount..enrich(): Fetches additional customer details from a REST API..to(): Sends enriched messages to a different topic.
5. Error Handling and Retry
Camel supports robust error handling:
onException(Exception.class)
.log("Error processing message: ${exception.message}")
.to("kafka:dead-letter-topic?brokers=localhost:9092")
.handled(true);
from("kafka:orders?brokers=localhost:9092")
.routeId("kafka-with-error-handling")
.unmarshal().json()
.process(exchange -> {
// Potentially failing transformation
Map<String, Object> body = exchange.getIn().getBody(Map.class);
if (body.get("amount") == null) {
throw new IllegalArgumentException("Amount is missing");
}
})
.marshal().json()
.to("kafka:validated-orders?brokers=localhost:9092");
6. Example: Integrating Kafka with a Database
You can consume Kafka messages and persist them directly to a database:
from("kafka:orders?brokers=localhost:9092")
.unmarshal().json()
.setBody(simple("INSERT INTO orders (id, customer, amount) VALUES (${body[id]}, '${body[customer]}', ${body[amount]})"))
.to("jdbc:dataSource");
Note: You must configure a DataSource in your Camel context.
7. Running Camel Routes
You can bootstrap Camel routes in many ways:
- Spring Boot (
camel-spring-boot-starter) - Quarkus (
camel-quarkus) - Standalone Java application
- Camel K (Kubernetes-native integration)
For a simple Java main class:
public class MainApp {
public static void main(String[] args) throws Exception {
Main main = new Main();
main.configure().addRoutesBuilder(new MyKafkaRoute());
main.run(args);
}
}
8. Resources and Further Reading
9. Conclusion
Apache Camel and Kafka together are a powerful duo for implementing event-driven architectures. Camel simplifies message routing, transformation, and integration with downstream systems, while Kafka provides a resilient, high-throughput backbone for events. With Camel’s declarative DSL, you can orchestrate complex workflows across microservices, databases, and external APIs—all in a maintainable, expressive way.
