Access control for Managed Service for Apache Kafka

This document describes how access control works in Managed Service for Apache Kafka.

Managed Kafka Service Agent

Managed Service for Apache Kafka uses a service agent to access Google Cloud resources. After you enable the Managed Kafka API, the Managed Kafka Service Agent is created.

The Managed Kafka Service Agent is a principal with the identifier service-PROJECT_NUMBER@gcp-sa-managedkafka.iam.gserviceaccount.com, where PROJECT_NUMBER is your Google Cloud project number.

This service agent requires the Managed Kafka Service Agent (roles/managedkafka.serviceAgent) IAM role on the project. This role is automatically granted when you enable the API. If you revoke this role, Managed Service for Apache Kafka can't create, update or delete clusters.

For some tasks, the Managed Kafka Service Agent requires additional roles. For example, if you use Kafka Connect, then you might need to grant the service agent permission to access sources and sinks, such as Pub/Sub or BigQuery.

Access control for clients

There are generally two ways to perform operations on Managed Service for Apache Kafka resources:

  • The Managed Kafka API. This option encompasses calling the REST APIs directly, as well as using Google Cloud CLI, the Google Cloud console, or the client libraries.

  • The open-source Apache Kafka API. Many open-source Kafka clients use this API.

Access control using the Managed Kafka API

When using the Managed Kafka API, authorization is performed by Identity and Access Management (IAM). To perform an operation on a resource, the principal must have the required Managed Kafka API permissions for that operation. The documentation for each task lists the IAM roles needed for that task. For example, to allow a principal to create Kafka topics, grant the principal the Managed Kafka Topic Editor role.

Access control using the Apache Kafka API

Clients that use the Apache Kafka API authenticate by using either SASL or mutual TLS (mTLS) authentication.

  • If a client connects with SASL, the authenticating principal needs the managedkafka.clusters.connect IAM permission. To give this permission, you can grant the Managed Kafka Client (roles/managedkafka.client) role on the principal.

  • If a client connects with mTLS, the managedkafka.clusters.connect permission isn't required. Instead, clients use client certificates to authenticate.

After the client connects, the service uses Kafka ACLs to authorize the client's operations, such as reading or altering resources.

Combined access control

For comprehensive security, you must configure the following:

  • IAM permissions for management access.

  • IAM permission to connect to the cluster, if clients use SASL.

  • Kafka ACLs for in-cluster data access and operations from open-source Apache Kafka clients, regardless of the authentication method.

For example, assume that you want to prevent a principal from editing topics. To do so, you would set up the following access controls:

  • Deny the principal the managedkafka.topics.update permission, to prevent the principal from using the Managed Kafka API to update topics (for example, by using the Google Cloud console).

  • Optionally, deny the principal the managedkafka.clusters.connect permission, to prevent the principal from connecting to the cluster with SASL authentication.

  • If the principal needs SASL access for other reasons, or if the principal connects using mTLS, create a Kafka ACL that denies ALTER, ALTER_CONFIGS, and DELETE operations on topics. For specific examples, see Deny modifying all topics.

Resource-level access control

To restrict a principal's IAM permissions to specific Managed Service for Apache Kafka resources within a project, such as individual clusters or topics, use IAM Conditions.

The following example shows an IAM condition that allows access only if the resource type is managedkafka.googleapis.com/Topic and the resource name ends in "test-topic":

{
    "expression": "resource.name.endsWith(\"topic-1\") &&\nresource.type == \"managedkafka.googleapis.com/Topic\"",
    "title": "topic-1",
    "description": "Grant access only to topic-1"
}

Some Managed Service for Apache Kafka requests, such as cluster creation or updates, return long-running operations. If you grant access at the cluster level, include access for all resources of type managedkafka.googleapis.com/Operation, in addition to the per-cluster resource condition. This permission ensures the principal can initiate the operation and monitor its progress.

What's next

Apache Kafka® is a registered trademark of The Apache Software Foundation or its affiliates in the United States and/or other countries.