手把手教你搭建 Kubernetes 监控体系：Prometheus + node-exporter + kube-state-metrics 完整部署指南

原创已于 2026-06-01 12:16:22 修改 · 327 阅读

5 ·

本内容遵循CC 4.0 BY-SA版权协议

GEO检测

收录于

Prometheus

k8s

于 2026-05-15 15:08:11 首次发布

前言

在 Kubernetes 集群中，监控是保障系统稳定性的关键环节。本文将从零开始，详细讲解如何部署Prometheus、node-exporter 和 kube-state-metrics，构建一套完整的 Kubernetes 监控体系。
涵盖以下内容：

创建命名空间
部署 node-exporter（DaemonSet）
部署 kube-state-metrics
配置 NFS 持久化存储
创建 Prometheus RBAC 权限
配置 Prometheus ConfigMap
部署 Prometheus StatefulSet
离线环境下的镜像导入方案

一、创建命名空间

kubectl create namespace monitoring

二、部署 node-exporter（DaemonSet）

node-exporter 以 DaemonSet 形式运行在每个节点上，采集节点硬件和操作系统指标。

cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
  labels:
    app: node-exporter
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      hostNetwork: true
      hostPID: true
      containers:
      - name: node-exporter
        image: prom/node-exporter:latest
        imagePullPolicy: IfNotPresent
        args:
        - --path.procfs=/host/proc
        - --path.sysfs=/host/sys
        ports:
        - containerPort: 9100
          hostPort: 9100
          name: metrics
        volumeMounts:
        - name: proc
          mountPath: /host/proc
        - name: sys
          mountPath: /host/sys
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys
      tolerations:
      - effect: NoSchedule
        operator: Exists
      - effect: NoExecute
        operator: Exists
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate
---
apiVersion: v1
kind: Service
metadata:
  name: node-exporter
  namespace: monitoring
  labels:
    app: node-exporter
spec:
  type: ClusterIP
  ports:
  - port: 9100
    targetPort: 9100
    name: metrics
  selector:
    app: node-exporter
EOF

三、部署 kube-state-metrics

kube-state-metrics 关注 Kubernetes 资源状态，如 Pod、Deployment、Service 等。

cat <<'EOF' | kubectl apply -f -
# ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-state-metrics
  namespace: monitoring

---
# ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-state-metrics
rules:
- apiGroups: [""]
  resources:
  - nodes
  - pods
  - services
  - resourcequotas
  - replicationcontrollers
  - limitranges
  - persistentvolumeclaims
  - persistentvolumes
  - namespaces
  - endpoints
  - configmaps
  verbs: ["list", "watch"]
- apiGroups: ["extensions"]
  resources:
  - daemonsets
  - deployments
  - replicasets
  - ingresses
  verbs: ["list", "watch"]
- apiGroups: ["apps"]
  resources:
  - daemonsets
  - deployments
  - replicasets
  - statefulsets
  verbs: ["list", "watch"]
- apiGroups: ["batch"]
  resources:
  - cronjobs
  - jobs
  verbs: ["list", "watch"]
- apiGroups: ["autoscaling"]
  resources:
  - horizontalpodautoscalers
  verbs: ["list", "watch"]
- apiGroups: ["networking.k8s.io"]
  resources:
  - ingresses
  verbs: ["list", "watch"]

---
# ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kube-state-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: monitoring

---
# Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: monitoring
  labels:
    app: kube-state-metrics
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-state-metrics
  template:
    metadata:
      labels:
        app: kube-state-metrics
    spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
        image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.10.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: http-metrics
        resources:
          limits:
            cpu: 200m
            memory: 256Mi
          requests:
            cpu: 100m
            memory: 128Mi

---
# Service
apiVersion: v1
kind: Service
metadata:
  name: kube-state-metrics
  namespace: monitoring
  labels:
    app: kube-state-metrics
spec:
  type: ClusterIP
  ports:
  - port: 8080
    targetPort: 8080
    name: http-metrics
  selector:
    app: kube-state-metrics
EOF

如果是国内服务器出现拉取镜像失败的报错，可以参考第九点

四、配置 NFS 持久化存储

4.1 在 NFS 服务端创建数据目录

我这里用k8s集群上的node1节点做nfs服务器

# 创建 prometheus 用户（UID 1003，注意执行前检查有没有这个id的用户）
useradd -u 1003 -m -s /bin/bash prometheus 2>/dev/null

# 创建数据目录
mkdir -p /data/nfs/prometheus

# 设置权限
chown -R prometheus:prometheus /data/nfs/prometheus
chmod -R 755 /data/nfs/prometheus

# 配置 NFS 导出
grep -q "/data/nfs/prometheus" /etc/exports || \
  echo "/data/nfs/prometheus *(rw,sync,no_subtree_check,no_root_squash)" >> /etc/exports

# 重新加载 NFS
exportfs -ra

# 验证
showmount -e localhost

4.2 创建 PV 和 PVC

# 创建 PV
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
  name: prometheus-data-pv
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: polpo-node1-prod
    path: /data/nfs/prometheus
EOF

# 创建 PVC
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-data
  namespace: monitoring
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  volumeName: prometheus-data-pv
EOF

# 验证绑定
kubectl get pv && kubectl get pvc -n monitoring

五、创建 Prometheus RBAC 权限

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus-server
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-server
rules:
- apiGroups: [""]
  resources: ["nodes", "nodes/proxy", "services", "endpoints", "pods"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus-server
subjects:
- kind: ServiceAccount
  name: prometheus-server
  namespace: monitoring
EOF

六、创建 Prometheus 配置 ConfigMap

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s

    scrape_configs:
      # 监控 Prometheus 自身
      - job_name: 'prometheus'
        static_configs:
          - targets: ['localhost:9090']
      
      # 采集 kube-state-metrics
      - job_name: 'kube-state-metrics'
        static_configs:
          - targets: ['kube-state-metrics:8080']
      
      # 采集 node-exporter - 使用 Endpoints 服务发现
      - job_name: 'kubernetes-nodes'
        kubernetes_sd_configs:
          - role: endpoints
        relabel_configs:
          - source_labels: [__meta_kubernetes_service_name]
            regex: 'node-exporter'
            action: keep
          - source_labels: [__meta_kubernetes_pod_node_name]
            target_label: host
          - source_labels: [__meta_kubernetes_pod_name]
            target_label: instance
          - source_labels: [__meta_kubernetes_pod_node_name]
            target_label: node
          - source_labels: [__meta_kubernetes_namespace]
            target_label: namespace
      
      # 采集 cAdvisor
      - job_name: 'kubernetes-cadvisor'
        kubernetes_sd_configs:
          - role: node
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: kubernetes.default.svc:443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
      
      # 采集 Pod 自定义指标
      - job_name: 'kubernetes-pods'
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
            action: replace
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: $1:$2
            target_label: __address__
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_pod_name]
            action: replace
            target_label: kubernetes_pod_name
EOF

七、创建 Headless Service

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitoring
spec:
  clusterIP: None
  ports:
  - name: http
    port: 9090
    targetPort: 9090
  selector:
    app: prometheus
EOF

八、创建 Prometheus StatefulSet

cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: prometheus
  namespace: monitoring
spec:
  replicas: 1
  serviceName: prometheus
  selector:
    matchLabels:
      app: prometheus
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus-server
      securityContext:
        fsGroup: 1003
        runAsUser: 1003
        runAsGroup: 1003
      initContainers:
        - name: fix-permissions
          image: docker.m.daocloud.io/library/busybox:latest
          command:
            - sh
            - -c
            - |
              chown -R 1003:1003 /prometheus
              chmod -R 755 /prometheus
          securityContext:
            runAsUser: 0
          volumeMounts:
            - name: prometheus-data
              mountPath: /prometheus
      containers:
        - name: prometheus
          image: docker.m.daocloud.io/prom/prometheus:latest
          imagePullPolicy: IfNotPresent
          args:
            - '--config.file=/etc/prometheus/prometheus.yml'
            - '--storage.tsdb.path=/prometheus'
            - '--storage.tsdb.retention.time=7d'
            - '--web.enable-lifecycle'
            - '--web.enable-admin-api'
            - '--web.listen-address=0.0.0.0:9090'
          ports:
            - containerPort: 9090
              name: http
          livenessProbe:
            httpGet:
              path: /-/healthy
              port: 9090
            initialDelaySeconds: 30
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /-/ready
              port: 9090
            initialDelaySeconds: 30
            periodSeconds: 10
          resources:
            limits:
              cpu: 2
              memory: 4Gi
            requests:
              cpu: 500m
              memory: 1Gi
          volumeMounts:
            - name: config
              mountPath: /etc/prometheus
            - name: prometheus-data
              mountPath: /prometheus
      volumes:
        - name: config
          configMap:
            name: prometheus-config
        - name: prometheus-data
          persistentVolumeClaim:
            claimName: prometheus-data
EOF

九、离线环境镜像导入方案

如果部署环境无法直接访问外网，可按以下步骤在离线环境下导入所需镜像。

9.1 在有网络的机器上下载并导出镜像

# 使用 ctr 拉取 kube-state-metrics
ctr image pull registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.10.0

# 使用 ctr 拉取 node-exporter
ctr image pull docker.io/prom/node-exporter:latest

# 导出 kube-state-metrics（压缩格式）
ctr image export --compress kube-state-metrics.tar.gz registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.10.0

# 导出 node-exporter（压缩格式）
ctr image export --compress node-exporter.tar.gz docker.io/prom/node-exporter:latest

9.2 将导出的文件传输到目标节点

scp kube-state-metrics.tar.gz node-exporter.tar.gz user@target-node:/path/to/images/

9.3 在目标节点导入镜像

# 导入镜像到 k8s.io 命名空间
ctr -n k8s.io image import kube-state-metrics.tar.gz
ctr -n k8s.io image import node-exporter.tar.gz

# 验证导入成功
ctr -n k8s.io image ls | grep -E "kube-state|node-exporter"

十、验证部署

10.1 查看所有资源状态

kubectl get all -n monitoring

10.2 访问 Prometheus UI

通过端口转发或配置ingress域名访问 Prometheus Web UI：

kubectl port-forward -n monitoring svc/prometheus 9090:9090

然后在浏览器访问 http://localhost:9090

10.3 检查 Targets 状态

在 Prometheus UI 中，点击 Status > Targets，确认所有监控目标状态均为 UP。

结语

至此，一套完整的 Kubernetes 监控体系已部署完成。通过 Prometheus 配合 node-exporter 和 kube-state-metrics，您可以：

监控集群节点的 CPU、内存、磁盘、网络等资源
监控 Kubernetes 资源对象的状态变化
通过 cAdvisor 获取容器级别的性能指标
为后续配置 Grafana 可视化展示和 Alertmanager 告警打下坚实基础

如有问题，欢迎在评论区留言交流！

标签

#kubernetes #prometheus #容器