k8s部署kafka集群外部访问（已验证并发布使用）

原创已于 2023-02-22 16:16:47 修改 · 3.9k 阅读 ·

本内容遵循CC 4.0 BY-SA版权协议

WinkingJay

关注

标签

#kafka #kubernetes #分布式

分类云计算与devops

于 2022-07-20 09:11:51 首次发布

k8s 专栏收录该内容

1 篇文章

订阅专栏

该配置文件展示了如何在Kubernetes环境中部署Zookeeper和Kafka集群。首先，通过创建Services和StatefulSets来设置Zookeeper集群，确保高可用性和客户端访问。接着，部署Kafka集群，配置了多个服务以暴露Kafka节点，并设置了StatefulSet来管理Kafka实例，同时连接到Zookeeper集群。所有组件都使用持久存储并指定了相应的端口和访问策略。

1.部署zookeeper集群

apiVersion: v1
kind: Service
metadata:
  labels:
    app: zookeeper-cluster
  namespace: kafka-outer
  name: zookeeper-cluster
spec:
  selector:
    app: zookeeper-cluster
  ports:
    - name: client
      port: 2181
      targetPort: 2181
    - name: follower
      port: 2888
      targetPort: 2888
    - name: leader
      port: 3888
      targetPort: 3888
  clusterIP: None
---
apiVersion: v1
kind: Service
metadata:
  namespace: kafka-outer
  name: zookeeper-cs
spec:
  selector:
    app: zookeeper-cluster
  type: NodePort
  ports:
    - name: client
      port: 2181
      nodePort: 30152
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: kafka-outer
  name: crs-zookeeper
spec:
  replicas: 3
  podManagementPolicy: Parallel
  serviceName: zookeeper-cluster
  selector:
    matchLabels:
      app: zookeeper-cluster
  template:
    metadata:
      labels:
        component: zookeeper-cluster
        app: zookeeper-cluster
    spec:
      containers:
        - name: zookeeper
          image: bitnami/zookeeper:latest
          imagePullPolicy: IfNotPresent
          securityContext:
            runAsUser: 0
          ports:
            - containerPort: 2181
            - containerPort: 2888
            - containerPort: 3888
          lifecycle:
            postStart:
              exec:
                command:
                  - "sh"
                  - "-c"
                  - >
                    echo $(( $(cat /etc/hosts | grep zookeeper | awk '{print($3)}' | awk '{split($0,array,"-")} END{print array[3]}') + 1 )) > /bitnami/zookeeper/data/myid
          env:
            - name: ALLOW_ANONYMOUS_LOGIN
              value: "yes"
            - name: ZOO_SERVERS
              value:  crs-zookeeper-0.zookeeper-cluster.kafka-outer.svc.cluster.local:2888:3888,crs-zookeeper-1.zookeeper-cluster.kafka-outer.svc.cluster.local:2888:3888,crs-zookeeper-2.zookeeper-cluster.kafka-outer.svc.cluster.local:2888:3888
          volumeMounts:
            - name: zoodata-outer
              mountPath: /bitnami/zookeeper
  volumeClaimTemplates:
    - metadata:
        name: zoodata-outer
      spec:
        storageClassName: common-nfs-storage    #指定为上面创建的 storageclass
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: 5Gi

2.部署kafka集群

#---
#apiVersion: v1
#kind: Service
#metadata:
#  namespace: kafka
#  name: kafka-cluster
#spec:
#  selector:
#    app: kafka-cluster
#  ports:
#    - port: 9092
#      targetPort: 9092
#  type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
  namespace: kafka-outer
  name: kafka-headless
spec:
  selector:
    app: kafka-cluster
  ports:
    - name: client
      port: 9092
      targetPort: 9092
  clusterIP: None
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-0
  namespace: kafka-outer
  labels:
    app: kafka-cluster
spec:
  ports:
    - port: 9092
      targetPort: 9092
      nodePort: 30127
      name: server
  type: NodePort
  selector:
    statefulset.kubernetes.io/pod-name: crs-kafka-0
#    app: kafka-cluster

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-1
  namespace: kafka-outer
  labels:
    app: kafka-cluster
spec:
  ports:
    - port: 9092
      targetPort: 9092
      nodePort: 30128
      name: server
  type: NodePort
  selector:
    statefulset.kubernetes.io/pod-name: crs-kafka-1

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-2
  namespace: kafka-outer
  labels:
    app: kafka-cluster
spec:
  ports:
    - port: 9092
      targetPort: 9092
      nodePort: 30129
      name: server
  type: NodePort
  selector:
    statefulset.kubernetes.io/pod-name: crs-kafka-2
---
apiVersion: apps/v1
kind: StatefulSet
#kind: Deployment
metadata:
  namespace: kafka-outer
  name: crs-kafka
spec:
  replicas: 3
  podManagementPolicy: Parallel
  serviceName: kafka-cluster
  selector:
    matchLabels:
      app: kafka-cluster
  template:
    metadata:
      labels:
        app: kafka-cluster
    spec:
      hostname: kafka
      containers:
        - name: kafka
          command:
            - bash
            - -ec
            - |
              HOSTNAME=`hostname -s`
              if [[ $HOSTNAME =~ (.*)-([0-9]+)$ ]]; then
                ORD=${BASH_REMATCH[2]}
                PORT=$((ORD + 30127))
                export KAFKA_CFG_ADVERTISED_LISTENERS="PLAINTEXT://192.168.0.147:$PORT"
              else
                echo "Failed to get index from hostname $HOST"
                exit 1
              fi
              exec /entrypoint.sh /run.sh
          image: bitnami/kafka:2.6.0
          #        image: bitnami/kafka:latest
          imagePullPolicy: IfNotPresent
          securityContext:
            runAsUser: 0
#          resources:
#            requests:
#              memory: "1G"
#              cpu: "0.5"
          ports:
            - containerPort: 9092
          env:
            - name: KAFKA_CFG_ZOOKEEPER_CONNECT
              value: crs-zookeeper-0.zookeeper-cluster.kafka-outer.svc.cluster.local:2181,crs-zookeeper-1.zookeeper-cluster.kafka-outer.svc.cluster.local:2181,crs-zookeeper-2.zookeeper-cluster.kafka-outer.svc.cluster.local:2181
            #          value: zookeeper-cluster:2181
            - name: ALLOW_PLAINTEXT_LISTENER
              value: "yes"
          volumeMounts:
            - name: kafkadata-outer
              mountPath: /bitnami/kafka
  volumeClaimTemplates:
    - metadata:
        name: kafkadata-outer
      spec:
        storageClassName: common-nfs-storage    #指定为上面创建的 storageclass
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: 5Gi