搭建zookeeper集群和kafka集群，SCRAM认证

原创已于 2025-03-14 08:46:54 修改 · 740 阅读 ·

本内容遵循CC 4.0 BY-SA版权协议

GEO检测

Nemo_ZR

关注

标签

#zookeeper #kafka #安全

分类大数据

于 2025-03-13 15:36:42 首次发布

服务器

hostname	ip
test01	192.168.12.101
test02	192.168.12.102
test03	192.168.12.103

软件版本

软件名	版本
linux	Rocky Linux 8.6 (Green Obsidian)
java	1.8.0_212
zookeeper	3.7.1
kafka	2.12-3.2.1

软件下载

略

软件安装

java，zookeeper，kafka都只需解压，解压后重命名一下

软件	目录
java	/opt/program/jdk
zookeeper	/opt/program/zookeeper
kafka	/opt/program/kafka

软件配置与启动

以下操作无特殊说明，都是需要在三台服务器上运行

系统相关配置

echo "192.168.12.101     test01" >> /etc/hosts
echo "192.168.12.102     test02" >> /etc/hosts
echo "192.168.12.103     test03" >> /etc/hosts

java配置

echo "export JAVA_HOME=/opt/program/jdk" >> /etc/profile
echo "export PATH=$PATH:$JAVA_HOME/bin" >> /etc/profile

source /etc/profile

zookeeper配置

新建/opt/program/zookeeper/conf/java.env

# 配置zookeeper启动的jvm参数，分配2G内存
export JVMFLAGS="-Xms2g -Xmx2g $JVMFLAGS"
# 加载认证文件
export JVMFLAGS="-Djava.security.auth.login.config=/opt/program/zookeeper/conf/zookeeper_jaas.conf $JVMFLAGS"

新建/opt/program/zookeeper/conf/zookeeper-env.sh

# 该文件为空
# 有的博主将认证环境放入这个文件中配置
# 我将认证放在java.env中，是同样的效果，只是和zookeeper的启动代码有些许差别
# 还有一个差别是如果将zookeeper的服务器间的认证、客户端和服务器间的两个认证分开的话
# 就要分别配置，在这个文件中，使用不同的参数

新建/opt/program/zookeeper/conf/zoo.cfg

tickTime=2000
initLimit=10
syncLimit=5
# 数据目录，后面的myid就会写入到这个目录中
dataDir=/opt/program/zookeeper/data
clientPort=2181
maxClientCnxns=100

# 暴露指标，没有这个需求的可以关闭这两个配置
metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
metricsProvider.httpPort=7000

# myid和这里的服务器是对应的
# server.myid=hostname:2888:3888
server.1=test01:2888:3888
server.2=test02:2888:3888
server.3=test03:2888:3888

# 配置客户端访问zookeeper，必须认证
zookeeper.sasl.client=true
sessionRequireClientSASLAuth=true
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
jaasLoginRenew=3600000

# zookeeper服务器之间认证
quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=true
quorum.auth.serverRequireSasl=true
quorum.auth.learner.loginContext=QuorumLearner
quorum.auth.server.loginContext=QuorumServer

新建/opt/program/zookeeper/conf/zookeeper_jaas.conf

QuorumServer {
        org.apache.zookeeper.server.auth.DigestLoginModule required
        username="zookeeper"
        password="abc123456"
        user_zookeeper="abc123456";
};

QuorumLearner {
        org.apache.zookeeper.server.auth.DigestLoginModule required
        username="zookeeper"
        password="abc123456"
        user_zookeeper="abc123456";
};


Server {
        org.apache.zookeeper.server.auth.DigestLoginModule required
        username="kafka"
        password="abc123456"
        user_kafka="abc123456";
};

Client {
        org.apache.zookeeper.server.auth.DigestLoginModule required
        username="kafka"
        password="abc123456"
        user_kafka="abc123456";
};

如果有新的组件要使用到zookeeper，那么就在Server和Client中增加账号，重启zookeeper集群

QuorumServer {
        org.apache.zookeeper.server.auth.DigestLoginModule required
        username="zookeeper"
        password="abc123456"
        user_zookeeper="abc123456";
};

QuorumLearner {
        org.apache.zookeeper.server.auth.DigestLoginModule required
        username="zookeeper"
        password="abc123456"
        user_zookeeper="abc123456";
};


Server {
        org.apache.zookeeper.server.auth.DigestLoginModule required
        username="kafka"
        password="abc123456"
        user_kafka="abc123456"
		user_dolphine="abc123456";
};

Client {
        org.apache.zookeeper.server.auth.DigestLoginModule required
        username="kafka"
        password="abc123456"
        user_kafka="abc123456"
		user_dolphine="abc123456";
};

新建/opt/program/zookeeper/data/myid

# 在test01上执行
echo 1 > /opt/program/zookeeper/data/myid

# 在test02上执行
echo 2 > /opt/program/zookeeper/data/myid

# 在test03上执行
echo 3 > /opt/program/zookeeper/data/myid

zookeeper启动

cd /opt/program/zookeeper
bin/zkServer.sh start

zookeeper状态验证

# 三台服务器都启动了zookeeper后，查看状态，会是leader或者是follower
cd /opt/program/zookeeper
bin/zkServer.sh stataus

zookeeper客户端验证

# 登入客户端
cd /opt/program/zookeeper
bin/zkCli.sh -server 192.168.12.101:2181

# 能登入进去，就说明客户端的认证已经通过
ls /
# 结果是 [zookeeper]

kafka配置

新建/opt/program/kafka/config/server.properties

# 三台服务器上，有两处是不同的，要分别配置
# 一个是broker.id=编号
# 一个是advertised.listeners=SASL_PLAINTEXT://主机IP:9092
# test01我给的编号是1，test02是2，test03是3
# 下面的是test01上的配置，另外两台机器上的配置，自行修改
broker.id=1
listeners=SASL_PLAINTEXT://0.0.0.0:9092
advertised.listeners=SASL_PLAINTEXT://192.168.12.101:9092
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
authorizer.class.name=kafka.security.authorizer.AclAuthorizer
allow.everyone.if.no.acl.found=false
super.users=User:admin
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/opt/program/kafka/data
num.partitions=3
default.replication.factor=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=3 
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.12.101:2181,192.168.12.102:2181,192.168.12.103:2181/kafka
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0

新建/opt/program/kafka/config/kafka_jaas.conf

Server和Client是kafka作为zookeeper客户端的认证凭证，与前面zookeeper配置要对应

KafkaServer {
    org.apache.kafka.common.security.scram.ScramLoginModule required
    username="admin"    
    password="abc123456";
};
KafkaClient {
    org.apache.kafka.common.security.scram.ScramLoginModule required
    username="admin"    
    password="abc123456";
};


Server {
        org.apache.zookeeper.server.auth.DigestLoginModule required
        username="kafka"
        password="abc123456"
        user_kafka="abc123456";
};
Client {
        org.apache.zookeeper.server.auth.DigestLoginModule required
        username="kafka"
        password="abc123456"
        user_kafka="abc123456";
};

修改/opt/program/kafka/bin/kafka-server-start.sh

sed -i '16i export KAFKA_OPTS=" -Djava.security.auth.login.config=/opt/program/kafka/config/kafka_jaas.conf " $KAFKA_OPTS' /opt/program/kafka/bin/kafka-server-start.sh

kakfa预启动

# 在任意一台机器上执行就可以了，例如在test01上
# 这个启动肯定会报错，但是连接到zookeeper上是不会报错的
# 可以按照zookeeper客户端验证一样，等会查看根目录会有一个kafka节点
cd /opt/program/kafka
sh bin/kafka-server-start.sh config/server.properties

kafka配置的zookeeper连接zookeeper.connect=192.168.12.101:2181,192.168.12.102:2181,192.168.12.103:2181/kafka

预启动的目的是在zookeeper中创建/kafka节点

如果说连接是

zookeeper.connect=192.168.12.101:2181,192.168.12.102:2181,192.168.12.103:2181

则不需要这一步

但是建议按照/kafka的方式，因为这样kafka的相关信息都会写入/kafka节点下，方便管理

如果有其他组件也要用，很容易混乱

kafka创建管理员账号

在任意一台机器上执行就可以了，例如在test01上

修改/opt/program/kafka/bin/kafka-configs.sh

sed -i '16i export KAFKA_OPTS=" -Djava.security.auth.login.config=/opt/program/kafka/config/kafka_jaas.conf " $KAFKA_OPTS' /opt/program/kafka/bin/kafka-configs.sh

创建admin账号

cd /opt/program/kafka
bin/kafka-configs.sh --zookeeper 192.168.12.101:2181/kafka --alter --add-config 'SCRAM-SHA-256=[iterations=8192,password=abc123456],SCRAM-SHA-512=[password=abc123456]' --entity-type users --entity-name admin

kafka启动

# 在三台服务器上执行
cd /opt/program/kafka
sh bin/kafka-server-start.sh -daemon config/server.properties

kafka查看日志

tail -100f logs/server.log

日志中没有报错，说明集群启动成功

kafka创建普通账号

如果上面创建admin账号时，在test01上，那么下面的命令也在test01上执行

cd /opt/program/kafka

# 创建producer账号
bin/kafka-configs.sh --zookeeper 192.168.12.101:2181/kafka --alter --add-config 'SCRAM-SHA-256=[iterations=8192,password=abc123456],SCRAM-SHA-512=[password=abc123456]' --entity-type users --entity-name producer

# 创建consumer账号
bin/kafka-configs.sh --zookeeper 192.168.12.101:2181/kafka --alter --add-config 'SCRAM-SHA-256=[iterations=8192,password=abc123456],SCRAM-SHA-512=[password=abc123456]' --entity-type users --entity-name producer

kafka创建主题

也在test01上执行

创建/opt/program/kafka/config/sasl_server.conf

security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="abc123456";

创建topic

cd /opt/program/kafka
# 创建topic
bin/kafka-topics.sh --bootstrap-server 192.168.12.101:9092 --command-config config/sasl_server.conf --create --partitions 3 --replication-factor 3 --topic test

#查看topic信息
bin/kafka-topics.sh --bootstrap-server 192.168.12.101:9092 --command-config config/sasl_server.conf --describe --topic test

kafka授权

授予producer账号，对主题test的producer权限

授予consumer账号，对主题test消费组test的consumer权限

cd /opt/program/kafka

# 授予producer权限
bin/kafka-acls.sh --bootstrap-server 192.168.12.101:9092 --command-config config/sasl_server.conf --add --allow-principal User:cn_app_record --producer --topic test

# 授予consumer权限
bin/kafka-acls.sh --bootstrap-server 192.168.12.101:9092 --command-config config/sasl_server.conf --add --allow-principal User:consumer --consumer --topic test --group test

kafka生产数据和消费数据

可以在多台服务器上执行，也可以只在一台服务器上执行，那就都放在test01上执行

创建/opt/program/kafka/config/sasl_producer.conf

security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="producer" password="abc123456";

创建/opt/program/kafka/config/sasl_consumer.conf

security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="consumer" password="abc123456";

启动生产者

cd /opt/program/kafka

bin/kafka-console-producer.sh --bootstrap-server 192.168.12.101:9092 --producer.config config/sasl_producer.conf --topic test

启动消费者

cd /opt/program/kafka

bin/kafka-console-consumer.sh --bootstrap-server 172.21.12.203:9092 --consumer.config config/sasl_consumer.conf --topic test --group test --from-beginning

在生产者的窗口随意输入字符，会在消费者窗口中打印出来，说明运行正常