Try Building a Meal-Planning Agent With Apache Kafka and Flink

Learn how to set up an event-driven multiagent AI system to simplify household duties with this step-by-step tutorial.

Mar 5th, 2025 9:00am by Sean Falconer

Featued image for: Try Building a Meal-Planning Agent With Apache Kafka and Flink

Image from Kaspars Grinvalds on Shutterstock

As a dad of two toddlers with very particular tastes — one constantly wants treats for dinner and the other refuses anything that isn’t beige — dinnertime at my house is a nightly episode of “Chopped: Toddler Edition.” Add in early bedtimes and the need to avoid meltdowns (theirs and mine) and dinner becomes less about gourmet aspirations and more about survival. The goal? Walk away with everyone fed, happy and preferably not covered in food. Meal planning is the ultimate balancing act: managing picky preferences, dietary needs and time constraints while trying to avoid cooking three separate meals. That’s why I turned to AI. A multiagent system, where each agent has a specific job, like managing preferences or optimizing grocery lists, is the perfect way to tackle this chaos. But just like wrangling two toddlers, coordination is key, and things can go sideways fast without a good plan. Enter event-driven architecture. It’s like the choreographed chaos of a well-run kitchen: Agents stay in sync, adapt to changes (like an unexpected snack raid) and recover gracefully from disasters (like burning the garlic bread). With tools like Kafka, Flink, LangChain and Claude, I’ve built an AI meal planner that takes the stress out of dinnertime, and gives me a fighting chance during those precious evening hours. In this article, I’ll share how I did it — and maybe even convince you to let AI take on your dinner drama. After all: If it can handle my kids, it can handle anything. Note: If you would like to just look at the code, jump over to my GitHub repo.

Why Event-Driven Multiagents?

At its core, an agent is like a little decision-making robot: It can analyze information, reason based on its environment and take actions.

Agent architecture (Inspired by https://arxiv.org/pdf/2304.03442).

What makes agents exciting is their ability to go beyond static, preprogrammed tasks. They can adapt to new inputs, learn from past experiences and handle dynamic problems — essentially doing the heavy lifting for repetitive or complex tasks. Whether it’s automating your workday or optimizing grocery shopping, agents turn reactive technology into proactive problem solvers.

Control logic, programmatic versus agentic.

Now, things start to get really interesting when you start to combine agents. You can combine multiple agents, each specializing in a different task, to solve a larger problem. That’s a multiagent system.

An example multiagent design pattern called the orchestrator-worker pattern.

Instead of a single jack-of-all-trades, you have a team of experts working together: One curates recipes, another calculates nutrition and yet another syncs with your family’s calendar. It’s collaboration at scale, unlocking solutions to problems that are too big or too messy for any one agent to handle alone. Of course, coordinating this team is its own challenge. Without a solid system, you’re looking at overlapping responsibilities, missed updates, or agents that fail to align with each other’s goals. Managing this is where event-driven design shines. In an event-driven architecture, agents don’t rely on rigid instructions. They respond to real-time events, like a change in schedules or a pantry update. Events act as a shared, dynamic language, letting agents stay synchronized, react quickly and handle failures without bringing the whole system down. This approach transforms multiagent systems from a theoretical concept into a practical tool. But how does this apply to meal planning for a busy family? Let’s explore how to design an event-driven multiagent meal planner.

Designing an Event-Driven Multiagent Meal Planner

Dinner at my house is a nightly puzzle: two picky toddlers with different preferences, my wife and I trying to eat healthier, and barely enough time to pull it all together before bedtime. For example, my son loves fish but my daughter is allergic, so every meal with fish needs a substitution for her. Meanwhile, they both adore macaroni and cheese, but my wife and I aim for meals with a better balance of protein, carbs and fats. Add in the constraints of a busy life and the problem starts to become stressful to deal with weekly when relying on manual planning. To tackle this, I designed a multiagent system — a team of AI experts, each specializing in a specific task. Here’s how it works:

Child meal-planning agent: This agent focuses on creating nutritious, kid-friendly meals that my toddlers will actually eat.
Adult meal-planning agent: This agent is an expert in planning meals for couples that are high in protein, low-glycemic and carb-conscious — but still tasty.
Shared preferences agent: Combines the child and adult meal plans into one cohesive menu, factoring in allergies and shared ingredients.
Format output agent: This agent takes the finalized meal plan, adds a grocery list and reformats it into a structured grocery list in JSON.

By breaking the work into specialized agents, the system improves accuracy and efficiency. Each agent operates like an expert chef for its domain, and the event-driven architecture keeps everything in sync without explicit dependencies.

Designing the User Interface

The web application is a standard three-tier architecture built with Next.js for the frontend and MongoDB as the application database. It’s intentionally kept simple and doesn’t include any direct AI logic or knowledge of Kafka. Its primary role is to let users configure their meal-planning settings, such as their kids’ likes and dislikes, and submit requests for new meal plans.

An example set of meal preferences.

When a new meal plan request is submitted, it’s written to MongoDB with a “processing” status. This triggers the multiagent workflow to generate a complete meal plan for the week. Once the process is finished, the status updates, and users can click on an individual meal plan to expand the UI. This reveals the full details for the week, including a breakdown of meals and an autogenerated grocery list. Meal Planner AI

Creating the Multiagent Workflow

To coordinate a multiagent system effectively, you need a shared language — a structured way for agents to exchange information, interpret commands and collaborate. In an event-driven architecture, events serve as this shared language, acting as structured updates that keep agents aligned, responsive and adaptable to change. Think of it as the system’s group chat, where agents broadcast updates, share context and perform tasks independently while staying synchronized. Kafka provides the backbone for this communication. Agents produce and consume events from Kafka topics, allowing them to react in real time to changes like new preferences or even new agents. Apache Flink adds the ability to process these streams, enabling complex reasoning and transformations on the fly. If you’re unfamiliar with Flink, it’s an open source stream processing framework built for handling large volumes of data in real time, and is ideal for high-throughput, low-latency applications. Flink is perfect for AI applications. With Confluent’s platform simplifying the management and scaling of this architecture, we can build a reliable, event-driven multiagent workflow that handles real-world complexities gracefully. The following diagram illustrates how I used these tools in Confluent Cloud to enable seamless and scalable collaboration across the agents.

Meal Planner AI architecture diagram.

Here’s how it works. When data is written to MongoDB, a source connector in Confluent Cloud creates an event in the Kafka topic “Meal Plan Requests.” This event starts the multiagent workflow. The child and adult meal-planning agents operate in parallel, generating their respective plans and producing events that are joined by Flink (as shown below) into a “Joined Preferences” topic.

CREATE TABLE `default`.`dev-us-east-1-cluster`.`meal-planner.output.joined-preferences` (
    `request_id` STRING,
    `child_preference` STRING,
    `adult_preference` STRING
) WITH (
  'value.format' = 'json-registry'
);

INSERT INTO `meal-planner.output.joined-preferences`
SELECT c.`request_id`, c.content as child_preference, a.content as adult_preference FROM `meal-planner.output.child-preferences` c
  JOIN `meal-planner.output.adult-preferences` a ON c.`request_id` = a.`request_id`;

As new events are written to the Joined Preferences topic, the shared meal plan agent is triggered, which consolidates the inputs into a unified family meal plan. The final event is written to another Kafka topic, activating the format output agent to structure the result into a user-friendly format, completing the workflow. This setup is highly scalable and decouples agent interactions, making it easy to add new agents or downstream consumers without affective performance or requiring changes to existing agents. Let’s dive deeper into the roles of the individual agents.

Creating a Meal Plan for Children and Adults

The child and adult meal-planning agents follow the “ReAct” design pattern. This design pattern combines reasoning and action into a single loop, enabling agents to make decisions and act iteratively based on their current understanding of a task.

Reasoning: The agent analyzes the situation, considers context and determines the next best action using its internal knowledge or external input.
Action: The agent performs the chosen action, which might involve interacting with the environment, querying data or generating output.
Feedback loop: The result of the action provides new information, which the agent incorporates into its reasoning for the next iteration.

This pattern is ideal for dynamic tasks requiring adaptability, as it allows the agent to continuously refine its approach and adjust to new developments in real time. I used LangGraph’s built-in support for this. The code of both the child and adult versions are quite similar, but the system prompt specifies a different behavior and set of expertise. Child meal-planning system prompt:

SYSTEM_PROMPT = """You are an expert at designing nutritious meals that toddlers love.
   You will be prompted to generate a weekly dinner plan.
   You'll have access to meal preferences. Use these as inspiration to come up with meals but you don't have to explicitly use these items.
   You'll have access to recent meals. Factor these in so you aren't repetitive.
   You must take into account any hard requirements about meals.
   Bias towards meals that can be made in less than 30 minutes. Keep meal preparation simple.
   There is no human in the loop, so don't prompt for additional input.
   """

Adult meal-planning system prompt:

SYSTEM_PROMPT = """You are an expert at designing high protein, low glycemic, low carb dinners for couples.
   You will be prompted to generate a weekly dinner plan.
   You'll have access to recent meals. Factor these in so you aren't repetitive.
   Bias towards meals that can be made in less than 30 minutes. Keep meal preparation simple.
   There is no human in the loop, so don't prompt for additional input.
   """

Both agents use tools. The child meal planner in particular uses several tools to take into account the meal planner’s configured settings. For example, the code below shows three tools for getting kid meal preferences, hard requirements and recent meals.

@tool
def get_kid_preferences():
   """Use this to get the likes and dislikes for the kids preferences."""
   # Connect to the MongoDB instance
   client = MongoClient(os.getenv("MONGODB_URI"))  # Replace with your MongoDB URI

   # Access the database and collection
   db = client['meal_planner']  # Database name
   collection = db['meal_preferences']  # Collection name

   projection = {"likes": 1, "dislikes": 1, "_id": 0}
   result = collection.find_one({}, projection)

   return result

@tool
def get_hard_requirements():
   """Use this to get the hard requirements for recommending a meal. These must be enforced."""
   # Connect to the MongoDB instance
   client = MongoClient(os.getenv("MONGODB_URI"))  # Replace with your MongoDB URI

   # Access the database and collection
   db = client['meal_planner']  # Database name
   collection = db['meal_preferences']  # Collection name

   projection = {"hardRequirements": 1, "_id": 0}
   result = collection.find_one({}, projection)

   return result

@tool
def get_recent_meals():
   """Use this to get recent meals."""
   # Connect to the MongoDB instance
   client = MongoClient(os.getenv("MONGODB_URI"))  # Replace with your MongoDB URI\
  
   # Access the database and collection
   db = client['meal_planner']
   collection = db['weekly_meal_plans']

   # Query to get the last two entries
   recent_meals = list(collection.find().sort([("$natural", -1)]).limit(2))
   return recent_meals

Creating a Shared Meal Plan

To merge the outputs of the child and adult meal-planning agents, I used the “reflection” design pattern, a framework that enables generative AI agents to evaluate and improve their outputs iteratively. This pattern operates through three key steps:

Generate: The agent produces an initial output based on its input or task.
Reflect: The agent critically evaluates the output, comparing it to task requirements or quality benchmarks. This step may involve self-assessment or using another agent for review.
Revise: Based on the reflection, the agent refines its output, producing a more accurate or polished result.

This pattern is particularly effective for complex tasks where a single-pass solution might fall short. By embedding reflection, agents autonomously improve their performance, ensuring higher reliability and quality without requiring external intervention.

Implementation

In the shared meal plan agent, I created two tailored prompts to guide the generation and reflection processes:

Generate content prompt:

generate_content_prompt = ChatPromptTemplate.from_messages(
   [
       (
           "system",
           "You are a meal planning assistant for families."
           "Your job is to combine the recommended meal plan for the children and the adults into a singular meal plan that works for the family."
           "Aim to minimize creating multiple dishes. Each meal should be able to work for both the adults and kids."
           "Make sure you include the same number of meals in the combined plan as in the original plans."
           "Output should contain the name of the meal, any modification or version for the children, any modification or version for the adults, core ingredients, prep time, and basic recipe."
           "If the user provides critique, respond with a revised version of your previous attempts.",
       ),
       MessagesPlaceholder(variable_name="messages"),
   ]
)

Reflection prompt:

reflection_prompt = ChatPromptTemplate.from_messages(
   [
       (
           "system",
           "You are a family meal planning expert grading the quality of the recommended meals on taste, variety, and nutritional value."
           "Generate critique and recommendations for the user's submission."
           "Provide detailed recommendations, including requests for greater variety, tastier meals, or higher nutrional value.",
       ),
       MessagesPlaceholder(variable_name="messages"),
   ]
)

Using LangGraph, I created a workflow graph connecting the generator node and reflection node. A conditional edge ensures the process iterates if the reflection step identifies areas for improvement. This iterative workflow dynamically refines the combined meal plan until it meets the desired standards, resulting in meals that balance the needs of both children and adults effectively.

builder = StateGraph(State)
builder.add_node("generate", generation_node)
builder.add_node("reflect", reflection_node)
builder.add_edge(START, "generate")

def should_continue(state: State):
   if len(state["messages"]) > MAX_ITERATIONS:
       return END
   return "reflect"

builder.add_conditional_edges("generate", should_continue)
builder.add_edge("reflect", "generate")
memory = MemorySaver()
graph = builder.compile(checkpointer=memory)

Formatting the Output

The final step in the workflow involves generating a comprehensive grocery list for the combined meal plan and formatting the output into a structured, user-friendly response. For this, I again applied the ReAct design pattern, equipping the agent with highly detailed instructions to ensure precise input interpretation and output generation. This agent features the most extensive and detailed system prompt in the entire workflow, as shown here. Once the structured response is generated, it is written back to a Kafka topic. A dedicated service consumes these messages and updates the corresponding weekly meal plan request in MongoDB. With this final update, the UI gains access to the fully completed meal plan and grocery list.

Things To Note on the Implementation

Both the agent services and the web application currently reside within the same repository, but they could easily be separated into distinct repositories for a more production-ready implementation. Similarly, while all the agents are part of a single project in this setup, they can be decoupled and deployed as independent serverless functions. This approach would allow each agent to operate autonomously, enhancing scalability, flexibility and fault isolation. Since I followed an event-driven approach, in the future I can extend or change the system of agents. For example, I might want to include a nutritional analysis agent that uses an external tool to calculate calories or macros based on the meal plan. Or I might want to incorporate human feedback to the meal plans, allowing the user to provide a thumbs up or thumbs down to meals so I can teach the AI about our preferences over time.

From Dinner Chaos to AI Harmony

Congratulations, you’ve made it through the whirlwind of AI meal planning, toddler tantrums and event-driven multiagent systems. My house may still occasionally resemble “Chopped: Toddler Edition,” but with Kafka, Flink and a set of AI agents, dinnertime has become a little less chaotic and a bit more enjoyable. The real takeaway here is how event-driven multiagent systems can simplify complex, real-world challenges. These agents don’t just work; they collaborate, adapt and scale effortlessly, all thanks to a shared “language” of events. It’s like having a kitchen staff that actually communicates and gets the job done without missing a beat. This architecture isn’t just for meal planning. It can be applied to any domain, from automating workflows at work to tackling everyday problems. For example, I’ve used a similar system to build a research agent and generate LinkedIn posts for me. Whether you’re automating your own tasks or building something bigger, the recipe remains the same: a solid design, multiagent collaboration and event-driven magic. Now, if you’ll excuse me, there’s a suspiciously quiet toddler in the other room …

Sean is the Head of AI at Confluent where he leads AI strategy and thought leadership. Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from AI to quantum computing. Sean...