Kubernetes operator（八） kubebuilder 实战演练之deploy-image插件的使用

原创 已于 2024-03-04 17:42:40 修改 · 1k 阅读 · 9 ·
本内容遵循CC 4.0 BY-SA版权协议
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
GEO检测
标签
#kubernetes #容器 #云原生 #云计算 #控制器
#Operator #Controller
于 2024-03-03 18:24:51 首次发布

云原生学习路线导航页（持续更新中）

1.案例需求

  • 我们期望开发一个Operator，能够实现如下效果：
    • 定义一个名为 Memcached 的 CR
    • 编写Controller具有调谐 Memcached 的能力
      • Memcached Spec中有一个Size，Controller 将不允许创建的Memcached实例 超过Size
      • Controller 还要负责更新资源的 Status

2.开发项目

2.1.创建一个项目Project

[root@master memcached-operator]# mkdir $GOPATH/memcached-operator
[root@master memcached-operator]# cd $GOPATH/memcached-operator
[root@master memcached-operator]# kubebuilder init --domain=graham924.com --repo=graham924.com/memcached-operator
INFO Writing kustomize manifests for you to edit...
INFO Writing scaffold for you to edit...
INFO Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.17.0
INFO Update dependencies:
$ go mod tidy
Next: define a resource with:
$ kubebuilder create api

2.2.创建一个API

这里 kubebuilder create api时，使用 deploy-image等 参数，直接就完成了全部代码的生成

2.2.1.deploy-image插件

  • 这里我们在使用 kubebuilder create api 时，将使用 deploy-image 的 plugin，帮我们自动生成一些代码。完整命令如下：
    kubebuilder create api --group cache --version v1alpha1 --kind Memcached --image=memcached:1.4.36-alpine --image-container-command="memcached,-m=64,-o,modern,-v" --image-container-port="11211" --run-as-user="1001" --plugins="deploy-image/v1-alpha" --make=false
  • 下面我会对这些参数一一进行解释：
    • kubebuilder create api：这是 Kubebuilder 工具的命令，用于创建一个新的 API 资源。
    • --group cache：指定新创建的 API 资源所属的 API 组，这里指定为 “cache”。
    • --version v1alpha1：指定新创建的 API 资源的版本号，这里指定为 “v1alpha1”。
    • --kind Memcached：指定新创建的资源的种类（Kind），这里指定为 “Memcached”，即自定义资源的类型为 Memcached。
    • --image=memcached:1.4.36-alpine：指定用于部署的镜像名称及版本，这里指定为 “memcached:1.4.36-alpine”。
    • --image-container-command="memcached,-m=64,-o,modern,-v"：指定容器的启动命令，这里设置为在容器中运行 memcached 服务，并指定了一些参数如内存限制、存储模式等。
    • --image-container-port="11211"：指定容器监听的端口号，这里指定为 “11211”，通常是 memcached 服务的默认端口。
    • --run-as-user="1001"：指定容器运行时的用户 ID，这里指定为 “1001”。
    • --plugins="deploy-image/v1-alpha"：指定要使用的插件，这里指定为部署镜像的插件 “deploy-image/v1-alpha”，用于将镜像部署到目标环境中。
    • --make=false：表示不立即构建二进制文件。当设置为 false 时，Kubebuilder 不会自动为你构建 Go 二进制文件，而是生成代码结构以供你自己构建。
  • deploy-image 的 学习
  • kubebuilder create api 时，使用 deploy-image plugin，会帮我们额外生成什么？
    • controllers/*_controller.go (脚手架controller的reconcile调谐逻辑框架)
    • controllers/*_controller_test.go (scaffold the tests for the controller)
    • controllers/*_suite_test.go (scaffold/update the suite of tests)
    • api//*_types.go (scaffold the specs for the new api)
    • config/samples/*_.yaml (scaffold default values for its CR)
    • main.go (update to add controller setup)
    • config/manager/manager.yaml (update with envvar to store the image)

2.2.2.创建API实践演示

  • 创建API
    [root@master memcached-operator]# kubebuilder create api --group cache --version v1alpha1 --kind Memcached --image=memcached:1.4.36-alpine --image-container-command="memcached,-m=64,-o,modern,-v" --image-container-port="11211" --run-as-user="1001" --plugins="deploy-image/v1-alpha" --make=false
INFO updating scaffold with deploy-image/v1alpha1 plugin...
INFO Writing scaffold for you to edit...
INFO Writing scaffold for you to edit...
INFO api/v1alpha1/memcached_types.go
INFO api/v1alpha1/groupversion_info.go
INFO internal/controller/suite_test.go
INFO internal/controller/memcached_controller.go
INFO internal/controller/memcached_controller_test.go
INFO Writing kustomize manifests for you to edit...
INFO api/v1alpha1/memcached_types.go
INFO config/samples/cache_v1alpha1_memcached.yaml
INFO internal/controller/memcached_controller.go
INFO creating import for % graham924.com/memcached-operator/api/v1alpha1
INFO internal/controller/memcached_controller_test.go
INFO creating import for % graham924.com/memcached-operator/api/v1alpha1
INFO Update dependencies:
$ go mod tidy
INFO Running make:
$ make manifests
mkdir -p /root/zgy/project/share-code-operator-study/memcached-operator/bin
Downloading sigs.k8s.io/controller-tools/cmd/controller-gen@v0.14.0
/root/zgy/project/share-code-operator-study/memcached-operator/bin/controller-gen-v0.14.0 rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
Next: check the implementation of your new API and controller. If you do changes in the API run the manifests with:
$ make manifests
  • 生成后的目录如下：
    [root@master memcached-operator]# tree
.
├── api
│   └── v1alpha1
│       ├── groupversion_info.go
│       └── memcached_types.go
├── bin
│   └── controller-gen-v0.14.0
├── cmd
│   └── main.go
├── config
│   ├── crd
│   │   ├── bases
│   │   │   └── cache.graham924.com_memcacheds.yaml
│   │   ├── kustomization.yaml
│   │   └── kustomizeconfig.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   └── manager_config_patch.yaml
│   ├── manager
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac
│   │   ├── auth_proxy_client_clusterrole.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   ├── leader_election_role.yaml
│   │   ├── memcached_editor_role.yaml
│   │   ├── memcached_viewer_role.yaml
│   │   ├── role_binding.yaml
│   │   ├── role.yaml
│   │   └── service_account.yaml
│   └── samples
│       ├── cache_v1alpha1_memcached.yaml
│       └── kustomization.yaml
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
├── internal
│   └── controller
│       ├── memcached_controller.go
│       ├── memcached_controller_test.go
│       └── suite_test.go
├── Makefile
├── PROJECT
├── README.md
└── test
    ├── e2e
    │   ├── e2e_suite_test.go
    │   └── e2e_test.go
    └── utils
        └── utils.go

18 directories, 41 files

2.2.3.查看CR的 types.go

  • 因为我们使用了 deploy-image 插件，所以 api/v1alpha1/memcached_types.go 的 Spec、Status 中，会默认生成几个字段
    • Spec 字段
      • Size：使用 +kubebuilder:validation 标记 设置Size的值在1~3之间。
      • ContainerPort：容器Port
    • Status 字段
      • Conditions：描述资源的状态信息
    ......
// MemcachedSpec defines the desired state of Memcached
type MemcachedSpec struct {
        // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
        // Important: Run "make" to regenerate code after modifying this file

        // Size defines the number of Memcached instances
        // The following markers will use OpenAPI v3 schema to validate the value
        // More info: https://book.kubebuilder.io/reference/markers/crd-validation.html
        // +kubebuilder:validation:Minimum=1
        // +kubebuilder:validation:Maximum=3
        // +kubebuilder:validation:ExclusiveMaximum=false
        Size int32 `json:"size,omitempty"`

        // Port defines the port that will be used to init the container with the image
        ContainerPort int32 `json:"containerPort,omitempty"`
}

// MemcachedStatus defines the observed state of Memcached
type MemcachedStatus struct {
        // Represents the observations of a Memcached's current state.
        // Memcached.status.conditions.type are: "Available", "Progressing", and "Degraded"
        // Memcached.status.conditions.status are one of True, False, Unknown.
        // Memcached.status.conditions.reason the 	value should be a CamelCase string and producers of specific
        // condition types may define expected values and meanings for this field, and whether the values
        // are considered a guaranteed API.
        // Memcached.status.conditions.Message is a human readable message indicating details about the transition.
        // For further information see: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties

        Conditions []metav1.Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type" protobuf:"bytes,1,rep,name=conditions"`
}
......
  • kubebuilder的官方文档中，给出了各种用于配置/代码生成的标记

2.2.4.查看CR的资源定义文件CRD

  • config/crd/bases/cache.example.com_memcacheds.yaml 文件
    [root@master memcached-operator]# cat config/crd/bases/cache.graham924.com_memcacheds.yaml
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.14.0
  name: memcacheds.cache.graham924.com
spec:
  group: cache.graham924.com
  names:
    kind: Memcached
    listKind: MemcachedList
    plural: memcacheds
    singular: memcached
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: Memcached is the Schema for the memcacheds API
        properties:
          apiVersion:
            description: |-
              APIVersion defines the versioned schema of this representation of an object.
              Servers should convert recognized schemas to the latest internal value, and
              may reject unrecognized values.
              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
            type: string
          kind:
            description: |-
              Kind is a string value representing the REST resource this object represents.
              Servers may infer this from the endpoint the client submits requests to.
              Cannot be updated.
              In CamelCase.
              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
            type: string
          metadata:
            type: object
          spec:
            description: MemcachedSpec defines the desired state of Memcached
            properties:
              containerPort:
                description: Port defines the port that will be used to init the container
                  with the image
                format: int32
                type: integer
              size:
                description: |-
                  Size defines the number of Memcached instances
                  The following markers will use OpenAPI v3 schema to validate the value
                  More info: https://book.kubebuilder.io/reference/markers/crd-validation.html
                format: int32
                maximum: 3
                minimum: 1
                type: integer
            type: object
          status:
            description: MemcachedStatus defines the observed state of Memcached
            properties:
              conditions:
                items:
                  description: "Condition contains details for one aspect of the current
                    state of this API Resource.\n---\nThis struct is intended for
                    direct use as an array at the field path .status.conditions.  For
                    example,\n\n\n\ttype FooStatus struct{\n\t    // Represents the
                    observations of a foo's current state.\n\t    // Known .status.conditions.type
                    are: \"Available\", \"Progressing\", and \"Degraded\"\n\t    //
                    +patchMergeKey=type\n\t    // +patchStrategy=merge\n\t    // +listType=map\n\t
                    \   // +listMapKey=type\n\t    Conditions []metav1.Condition `json:\"conditions,omitempty\"
                    patchStrategy:\"merge\" patchMergeKey:\"type\" protobuf:\"bytes,1,rep,name=conditions\"`\n\n\n\t
                    \   // other fields\n\t}"
                  properties:
                    lastTransitionTime:
                      description: |-
                        lastTransitionTime is the last time the condition transitioned from one status to another.
                        This should be when the underlying condition changed.  If that is not known, then using the time when the API field changed is acceptable.
                      format: date-time
                      type: string
                    message:
                      description: |-
                        message is a human readable message indicating details about the transition.
                        This may be an empty string.
                      maxLength: 32768
                      type: string
                    observedGeneration:
                      description: |-
                        observedGeneration represents the .metadata.generation that the condition was set based upon.
                        For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
                        with respect to the current state of the instance.
                      format: int64
                      minimum: 0
                      type: integer
                    reason:
                      description: |-
                        reason contains a programmatic identifier indicating the reason for the condition's last transition.
                        Producers of specific condition types may define expected values and meanings for this field,
                        and whether the values are considered a guaranteed API.
                        The value should be a CamelCase string.
                        This field may not be empty.
                      maxLength: 1024
                      minLength: 1
                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
                      type: string
                    status:
                      description: status of the condition, one of True, False, Unknown.
                      enum:
                      - "True"
                      - "False"
                      - Unknown
                      type: string
                    type:
                      description: |-
                        type of condition in CamelCase or in foo.example.com/CamelCase.
                        ---
                        Many .condition.type values are consistent across resources like Available, but because arbitrary conditions can be
                        useful (see .node.status.conditions), the ability to deconflict is important.
                        The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
                      maxLength: 316
                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
                      type: string
                  required:
                  - lastTransitionTime
                  - message
                  - reason
                  - status
                  - type
                  type: object
                type: array
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}
  • 可以看出，+kubebuilder 标记的内容，会展示在CRD的定义文件中的，下面以Spec的内容为例，Status的内容也是一样的
    spec:
  description: MemcachedSpec defines the desired state of Memcached
  properties:
    containerPort:
      description: Port defines the port that will be used to init the container
        with the image
      format: int32
      type: integer
    size:
      description: |-
        Size defines the number of Memcached instances
        The following markers will use OpenAPI v3 schema to validate the value
        More info: https://book.kubebuilder.io/reference/markers/crd-validation.html
      format: int32
      maximum: 3
      minimum: 1
      type: integer
  type: object

2.2.5.查看CR yaml示例

  • config/crd/bases/cache.example.com_memcacheds.yaml 文件
    apiVersion: cache.graham924.com/v1alpha1
kind: Memcached
metadata:
  name: memcached-sample
spec:
  # TODO(user): edit the following value to ensure the number
  # of Pods/Instances your Operand must have on cluster
  size: 1

  # TODO(user): edit the following value to ensure the container has the right port to be initialized
  containerPort: 11211

2.2.6.查看CR的 controller.go

  • 因为我们使用了 deploy-image 插件，所以 internal/controller/memcached_controller.go 的 Reconcile 已经给生成了很多代码逻辑
    /*
Copyright 2024.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package controller

import (
        "context"
        "fmt"
        "os"
        "strings"
        "time"

        appsv1 "k8s.io/api/apps/v1"
        corev1 "k8s.io/api/core/v1"
        apierrors "k8s.io/apimachinery/pkg/api/errors"
        "k8s.io/apimachinery/pkg/api/meta"
        metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
        "k8s.io/apimachinery/pkg/runtime"
        "k8s.io/apimachinery/pkg/types"
        "k8s.io/client-go/tools/record"
        ctrl "sigs.k8s.io/controller-runtime"
        "sigs.k8s.io/controller-runtime/pkg/client"
        "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
        "sigs.k8s.io/controller-runtime/pkg/log"

        cachev1alpha1 "graham924.com/memcached-operator/api/v1alpha1"
)

const memcachedFinalizer = "cache.graham924.com/finalizer"

// Definitions to manage status conditions
const (
        // typeAvailableMemcached represents the status of the Deployment reconciliation
        typeAvailableMemcached = "Available"
        // typeDegradedMemcached represents the status used when the custom resource is deleted and the finalizer operations are must to occur.
        typeDegradedMemcached = "Degraded"
)

// MemcachedReconciler reconciles a Memcached object
type MemcachedReconciler struct {
        client.Client
        Scheme   *runtime.Scheme
        Recorder record.EventRecorder
}

// The following markers are used to generate the rules permissions (RBAC) on config/rbac using controller-gen
// when the command <make manifests> is executed.
// To know more about markers see: https://book.kubebuilder.io/reference/markers.html

//+kubebuilder:rbac:groups=cache.graham924.com,resources=memcacheds,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=cache.graham924.com,resources=memcacheds/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=cache.graham924.com,resources=memcacheds/finalizers,verbs=update
//+kubebuilder:rbac:groups=core,resources=events,verbs=create;patch
//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;watch

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// It is essential for the controller's reconciliation loop to be idempotent. By following the Operator
// pattern you will create Controllers which provide a reconcile function
// responsible for synchronizing resources until the desired state is reached on the cluster.
// Breaking this recommendation goes against the design principles of controller-runtime.
// and may lead to unforeseen consequences such as resources becoming stuck and requiring manual intervention.
// For further info:
// - About Operator Pattern: https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
// - About Controllers: https://kubernetes.io/docs/concepts/architecture/controller/
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.17.0/pkg/reconcile
func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
        log := log.FromContext(ctx)

        // Fetch the Memcached instance
        // The purpose is check if the Custom Resource for the Kind Memcached
        // is applied on the cluster if not we return nil to stop the reconciliation
        memcached := &cachev1alpha1.Memcached{}
        err := r.Get(ctx, req.NamespacedName, memcached)
        if err != nil {
                if apierrors.IsNotFound(err) {
                        // If the custom resource is not found then, it usually means that it was deleted or not created
                        // In this way, we will stop the reconciliation
                        log.Info("memcached resource not found. Ignoring since object must be deleted")
                        return ctrl.Result{}, nil
                }
                // Error reading the object - requeue the request.
                log.Error(err, "Failed to get memcached")
                return ctrl.Result{}, err
        }

        // Let's just set the status as Unknown when no status are available
        if memcached.Status.Conditions == nil || len(memcached.Status.Conditions) == 0 {
                meta.SetStatusCondition(&memcached.Status.Conditions, metav1.Condition{Type: typeAvailableMemcached, Status: metav1.ConditionUnknown, Reason: "Reconciling", Message: "Starting reconciliation"})
                if err = r.Status().Update(ctx, memcached); err != nil {
                        log.Error(err, "Failed to update Memcached status")
                        return ctrl.Result{}, err
                }

                // Let's re-fetch the memcached Custom Resource after update the status
                // so that we have the latest state of the resource on the cluster and we will avoid
                // raise the issue "the object has been modified, please apply
                // your changes to the latest version and try again" which would re-trigger the reconciliation
                // if we try to update it again in the following operations
                if err := r.Get(ctx, req.NamespacedName, memcached); err != nil {
                        log.Error(err, "Failed to re-fetch memcached")
                        return ctrl.Result{}, err
                }
        }

        // Let's add a finalizer. Then, we can define some operations which should
        // occurs before the custom resource to be deleted.
        // More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers
        if !controllerutil.ContainsFinalizer(memcached, memcachedFinalizer) {
                log.Info("Adding Finalizer for Memcached")
                if ok := controllerutil.AddFinalizer(memcached, memcachedFinalizer); !ok {
                        log.Error(err, "Failed to add finalizer into the custom resource")
                        return ctrl.Result{Requeue: true}, nil
                }

                if err = r.Update(ctx, memcached); err != nil {
                        log.Error(err, "Failed to update custom resource to add finalizer")
                        return ctrl.Result{}, err
                }
        }

        // Check if the Memcached instance is marked to be deleted, which is
        // indicated by the deletion timestamp being set.
        isMemcachedMarkedToBeDeleted := memcached.GetDeletionTimestamp() != nil
        if isMemcachedMarkedToBeDeleted {
                if controllerutil.ContainsFinalizer(memcached, memcachedFinalizer) {
                        log.Info("Performing Finalizer Operations for Memcached before delete CR")

                        // Let's add here an status "Downgrade" to define that this resource begin its process to be terminated.
                        meta.SetStatusCondition(&memcached.Status.Conditions, metav1.Condition{Type: typeDegradedMemcached,
                                Status: metav1.ConditionUnknown, Reason: "Finalizing",
                                Message: fmt.Sprintf("Performing finalizer operations for the custom resource: %s ", memcached.Name)})

                        if err := r.Status().Update(ctx, memcached); err != nil {
                                log.Error(err, "Failed to update Memcached status")
                                return ctrl.Result{}, err
                        }

                        // Perform all operations required before remove the finalizer and allow
                        // the Kubernetes API to remove the custom resource.
                        r.doFinalizerOperationsForMemcached(memcached)

                        // TODO(user): If you add operations to the doFinalizerOperationsForMemcached method
                        // then you need to ensure that all worked fine before deleting and updating the Downgrade status
                        // otherwise, you should requeue here.

                        // Re-fetch the memcached Custom Resource before update the status
                        // so that we have the latest state of the resource on the cluster and we will avoid
                        // raise the issue "the object has been modified, please apply
                        // your changes to the latest version and try again" which would re-trigger the reconciliation
                        if err := r.Get(ctx, req.NamespacedName, memcached); err != nil {
                                log.Error(err, "Failed to re-fetch memcached")
                                return ctrl.Result{}, err
                        }

                        meta.SetStatusCondition(&memcached.Status.Conditions, metav1.Condition{Type: typeDegradedMemcached,
                                Status: metav1.ConditionTrue, Reason: "Finalizing",
                                Message: fmt.Sprintf("Finalizer operations for custom resource %s name were successfully accomplished", memcached.Name)})

                        if err := r.Status().Update(ctx, memcached); err != nil {
                                log.Error(err, "Failed to update Memcached status")
                                return ctrl.Result{}, err
                        }

                        log.Info("Removing Finalizer for Memcached after successfully perform the operations")
                        if ok := controllerutil.RemoveFinalizer(memcached, memcachedFinalizer); !ok {
                                log.Error(err, "Failed to remove finalizer for Memcached")
                                return ctrl.Result{Requeue: true}, nil
                        }

                        if err := r.Update(ctx, memcached); err != nil {
                                log.Error(err, "Failed to remove finalizer for Memcached")
                                return ctrl.Result{}, err
                        }
                }
                return ctrl.Result{}, nil
        }

        // Check if the deployment already exists, if not create a new one
        found := &appsv1.Deployment{}
        err = r.Get(ctx, types.NamespacedName{Name: memcached.Name, Namespace: memcached.Namespace}, found)
        if err != nil && apierrors.IsNotFound(err) {
                // Define a new deployment
                dep, err := r.deploymentForMemcached(memcached)
                if err != nil {
                        log.Error(err, "Failed to define new Deployment resource for Memcached")

                        // The following implementation will update the status
                        meta.SetStatusCondition(&memcached.Status.Conditions, metav1.Condition{Type: typeAvailableMemcached,
                                Status: metav1.ConditionFalse, Reason: "Reconciling",
                                Message: fmt.Sprintf("Failed to create Deployment for the custom resource (%s): (%s)", memcached.Name, err)})

                        if err := r.Status().Update(ctx, memcached); err != nil {
                                log.Error(err, "Failed to update Memcached status")
                                return ctrl.Result{}, err
                        }

                        return ctrl.Result{}, err
                }

                log.Info("Creating a new Deployment",
                        "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
                if err = r.Create(ctx, dep); err != nil {
                        log.Error(err, "Failed to create new Deployment",
                                "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
                        return ctrl.Result{}, err
                }

                // Deployment created successfully
                // We will requeue the reconciliation so that we can ensure the state
                // and move forward for the next operations
                return ctrl.Result{RequeueAfter: time.Minute}, nil
        } else if err != nil {
                log.Error(err, "Failed to get Deployment")
                // Let's return the error for the reconciliation be re-trigged again
                return ctrl.Result{}, err
        }

        // The CRD API is defining that the Memcached type, have a MemcachedSpec.Size field
        // to set the quantity of Deployment instances is the desired state on the cluster.
        // Therefore, the following code will ensure the Deployment size is the same as defined
        // via the Size spec of the Custom Resource which we are reconciling.
        size := memcached.Spec.Size
        if *found.Spec.Replicas != size {
                found.Spec.Replicas = &size
                if err = r.Update(ctx, found); err != nil {
                        log.Error(err, "Failed to update Deployment",
                                "Deployment.Namespace", found.Namespace, "Deployment.Name", found.Name)

                        // Re-fetch the memcached Custom Resource before update the status
                        // so that we have the latest state of the resource on the cluster and we will avoid
                        // raise the issue "the object has been modified, please apply
                        // your changes to the latest version and try again" which would re-trigger the reconciliation
                        if err := r.Get(ctx, req.NamespacedName, memcached); err != nil {
                                log.Error(err, "Failed to re-fetch memcached")
                                return ctrl.Result{}, err
                        }

                        // The following implementation will update the status
                        meta.SetStatusCondition(&memcached.Status.Conditions, metav1.Condition{Type: typeAvailableMemcached,
                                Status: metav1.ConditionFalse, Reason: "Resizing",
                                Message: fmt.Sprintf("Failed to update the size for the custom resource (%s): (%s)", memcached.Name, err)})

                        if err := r.Status().Update(ctx, memcached); err != nil {
                                log.Error(err, "Failed to update Memcached status")
                                return ctrl.Result{}, err
                        }

                        return ctrl.Result{}, err
                }

                // Now, that we update the size we want to requeue the reconciliation
                // so that we can ensure that we have the latest state of the resource before
                // update. Also, it will help ensure the desired state on the cluster
                return ctrl.Result{Requeue: true}, nil
        }

        // The following implementation will update the status
        meta.SetStatusCondition(&memcached.Status.Conditions, metav1.Condition{Type: typeAvailableMemcached,
                Status: metav1.ConditionTrue, Reason: "Reconciling",
                Message: fmt.Sprintf("Deployment for custom resource (%s) with %d replicas created successfully", memcached.Name, size)})

        if err := r.Status().Update(ctx, memcached); err != nil {
                log.Error(err, "Failed to update Memcached status")
                return ctrl.Result{}, err
        }

        return ctrl.Result{}, nil
}

// finalizeMemcached will perform the required operations before delete the CR.
func (r *MemcachedReconciler) doFinalizerOperationsForMemcached(cr *cachev1alpha1.Memcached) {
        // TODO(user): Add the cleanup steps that the operator
        // needs to do before the CR can be deleted. Examples
        // of finalizers include performing backups and deleting
        // resources that are not owned by this CR, like a PVC.

        // Note: It is not recommended to use finalizers with the purpose of delete resources which are
        // created and managed in the reconciliation. These ones, such as the Deployment created on this reconcile,
        // are defined as depended of the custom resource. See that we use the method ctrl.SetControllerReference.
        // to set the ownerRef which means that the Deployment will be deleted by the Kubernetes API.
        // More info: https://kubernetes.io/docs/tasks/administer-cluster/use-cascading-deletion/

        // The following implementation will raise an event
        r.Recorder.Event(cr, "Warning", "Deleting",
                fmt.Sprintf("Custom Resource %s is being deleted from the namespace %s",
                        cr.Name,
                        cr.Namespace))
}

// deploymentForMemcached returns a Memcached Deployment object
func (r *MemcachedReconciler) deploymentForMemcached(
        memcached *cachev1alpha1.Memcached) (*appsv1.Deployment, error) {
        ls := labelsForMemcached(memcached.Name)
        replicas := memcached.Spec.Size

        // Get the Operand image
        image, err := imageForMemcached()
        if err != nil {
                return nil, err
        }

        dep := &appsv1.Deployment{
                ObjectMeta: metav1.ObjectMeta{
                        Name:      memcached.Name,
                        Namespace: memcached.Namespace,
                },
                Spec: appsv1.DeploymentSpec{
                        Replicas: &replicas,
                        Selector: &metav1.LabelSelector{
                                MatchLabels: ls,
                        },
                        Template: corev1.PodTemplateSpec{
                                ObjectMeta: metav1.ObjectMeta{
                                        Labels: ls,
                                },
                                Spec: corev1.PodSpec{
                                        // TODO(user): Uncomment the following code to configure the nodeAffinity expression
                                        // according to the platforms which are supported by your solution. It is considered
                                        // best practice to support multiple architectures. build your manager image using the
                                        // makefile target docker-buildx. Also, you can use docker manifest inspect <image>
                                        // to check what are the platforms supported.
                                        // More info: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity
                                        //Affinity: &corev1.Affinity{
                                        //      NodeAffinity: &corev1.NodeAffinity{
                                        //              RequiredDuringSchedulingIgnoredDuringExecution: &corev1.NodeSelector{
                                        //                      NodeSelectorTerms: []corev1.NodeSelectorTerm{
                                        //                              {
                                        //                                      MatchExpressions: []corev1.NodeSelectorRequirement{
                                        //                                              {
                                        //                                                      Key:      "kubernetes.io/arch",
                                        //                                                      Operator: "In",
                                        //                                                      Values:   []string{"amd64", "arm64", "ppc64le", "s390x"},
                                        //                                              },
                                        //                                              {
                                        //                                                      Key:      "kubernetes.io/os",
                                        //                                                      Operator: "In",
                                        //                                                      Values:   []string{"linux"},
                                        //                                              },
                                        //                                      },
                                        //                              },
                                        //                      },
                                        //              },
                                        //      },
                                        //},
                                        SecurityContext: &corev1.PodSecurityContext{
                                                RunAsNonRoot: &[]bool{true}[0],
                                                // IMPORTANT: seccomProfile was introduced with Kubernetes 1.19
                                                // If you are looking for to produce solutions to be supported
                                                // on lower versions you must remove this option.
                                                SeccompProfile: &corev1.SeccompProfile{
                                                        Type: corev1.SeccompProfileTypeRuntimeDefault,
                                                },
                                        },
                                        Containers: []corev1.Container{{
                                                Image:           image,
                                                Name:            "memcached",
                                                ImagePullPolicy: corev1.PullIfNotPresent,
                                                // Ensure restrictive context for the container
                                                // More info: https://kubernetes.io/docs/concepts/security/pod-security-standards/#restricted
                                                SecurityContext: &corev1.SecurityContext{
                                                        RunAsNonRoot:             &[]bool{true}[0],
                                                        RunAsUser:                &[]int64{1001}[0],
                                                        AllowPrivilegeEscalation: &[]bool{false}[0],
                                                        Capabilities: &corev1.Capabilities{
                                                                Drop: []corev1.Capability{
                                                                        "ALL",
                                                                },
                                                        },
                                                },
                                                Ports: []corev1.ContainerPort{{
                                                        ContainerPort: memcached.Spec.ContainerPort,
                                                        Name:          "memcached",
                                                }},
                                                Command: []string{"memcached", "-m=64", "-o", "modern", "-v"},
                                        }},
                                },
                        },
                },
        }

        // Set the ownerRef for the Deployment
        // More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/
        if err := ctrl.SetControllerReference(memcached, dep, r.Scheme); err != nil {
                return nil, err
        }
        return dep, nil
}

// labelsForMemcached returns the labels for selecting the resources
// More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/
func labelsForMemcached(name string) map[string]string {
        var imageTag string
        image, err := imageForMemcached()
        if err == nil {
                imageTag = strings.Split(image, ":")[1]
        }
        return map[string]string{"app.kubernetes.io/name": "Memcached",
                "app.kubernetes.io/instance":   name,
                "app.kubernetes.io/version":    imageTag,
                "app.kubernetes.io/part-of":    "memcached-operator",
                "app.kubernetes.io/created-by": "controller-manager",
        }
}

// imageForMemcached gets the Operand image which is managed by this controller
// from the MEMCACHED_IMAGE environment variable defined in the config/manager/manager.yaml
func imageForMemcached() (string, error) {
        var imageEnvVar = "MEMCACHED_IMAGE"
        image, found := os.LookupEnv(imageEnvVar)
        if !found {
                return "", fmt.Errorf("Unable to find %s environment variable with the image", imageEnvVar)
        }
        return image, nil
}

// SetupWithManager sets up the controller with the Manager.
// Note that the Deployment will be also watched in order to ensure its
// desirable state on the cluster
func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
        return ctrl.NewControllerManagedBy(mgr).
                For(&cachev1alpha1.Memcached{}).
                Owns(&appsv1.Deployment{}).
                Complete(r)
}

2.2.7.controller.go 中 SetupWithManager 方法

  • SetupWithManager 方法中，已经为我们的CR创建ListWatch机制了
    // SetupWithManager sets up the controller with the Manager.
// Note that the Deployment will be also watched in order to ensure its
// desirable state on the cluster
func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
        return ctrl.NewControllerManagedBy(mgr).
                For(&cachev1alpha1.Memcached{}).
                Owns(&appsv1.Deployment{}).
                Complete(r)
}
  • Manager 是controller-runtime提供的组件，用于监督和管理 Controller，我们后续会详细学习。
  • SetupWithManager 方法就是将 当前Controller注册到 Manager 中

2.2.8.controller.go 中 设置deployment 的 ownerRef

  • deploymentForMemcached 中，创建的deployment，ownerRef 已经给设置成了相应的Memcached资源
    // deploymentForMemcached returns a Memcached Deployment object
func (r *MemcachedReconciler) deploymentForMemcached(
        memcached *cachev1alpha1.Memcached) (*appsv1.Deployment, error) {
        ls := labelsForMemcached(memcached.Name)
        replicas := memcached.Spec.Size
	
	......

		if err := ctrl.SetControllerReference(memcached, dep, r.Scheme); err != nil {
                return nil, err
        }
        return dep, nil
}

2.2.9.controller.go 中 rbac 的 kubebuilder 标记

  • Reconcile 方法上方，使用 rbac 的 kubebuilder 标记，标记了当前Operator的访问权限
  • 当你修改 controller.go 里的rbac标记后，需要执行 make manifests 或 make generate 更新 config/rbac 目录下的rbac权限文件
    //+kubebuilder:rbac:groups=cache.graham924.com,resources=memcacheds,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=cache.graham924.com,resources=memcacheds/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=cache.graham924.com,resources=memcacheds/finalizers,verbs=update
//+kubebuilder:rbac:groups=core,resources=events,verbs=create;patch
//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;watch

3.CRD 及 Controller 部署

3.1.安装CRD

[root@master memcached-operator]# make install
/root/zgy/project/share-code-operator-study/memcached-operator/bin/controller-gen-v0.14.0 rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
Downloading sigs.k8s.io/kustomize/kustomize/v5@v5.3.0
/root/zgy/project/share-code-operator-study/memcached-operator/bin/kustomize-v5.3.0 build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/memcacheds.cache.graham924.com created

3.2.打包Controller并上传镜像

  • cd 到 memcached-operator 的所在目录，修改 Dockerfile 文件，加上这么两句
    # Build the manager binary
FROM golang:1.21 AS builder
ARG TARGETOS
ARG TARGETARCH

# 就是加上这两句，设置一下go的国内代理加速
ENV GO111MODULE=on
ENV GOPROXY=https://goproxy.cn

......
  • 然后 修改Makefile 文件，在docker-build命令中，添加 --network host ，这是让我们的机器使用主机网络，能够连接外网
    # If you wish to build the manager image targeting other platforms you can use the --platform flag.
# (i.e. docker build --platform linux/arm64). However, you must enable docker buildKit for it.
# More info: https://docs.docker.com/develop/develop-images/build_enhancements/
.PHONY: docker-build
docker-build: ## Build docker image with the manager.
        $(CONTAINER_TOOL) build --network host -t ${IMG} .
  • 执行打包上传命令
    make docker-build docker-push IMG=gesang321/memcached-operator:v1alpha1

3.3.部署Controller

  • 部署Controller
    [root@master memcached-operator]# make deploy IMG=gesang321/memcached-operator:v1alpha1
/root/zgy/project/share-code-operator-study/memcached-operator/bin/controller-gen-v0.14.0 rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
cd config/manager && /root/zgy/project/share-code-operator-study/memcached-operator/bin/kustomize-v5.3.0 edit set image controller=gesang321/memcached-operator:v1alpha1
/root/zgy/project/share-code-operator-study/memcached-operator/bin/kustomize-v5.3.0 build config/default | kubectl apply -f -
namespace/memcached-operator-system created
customresourcedefinition.apiextensions.k8s.io/memcacheds.cache.graham924.com created
serviceaccount/memcached-operator-controller-manager created
role.rbac.authorization.k8s.io/memcached-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/memcached-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/memcached-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/memcached-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/memcached-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/memcached-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/memcached-operator-proxy-rolebinding created
service/memcached-operator-controller-manager-metrics-service created
deployment.apps/memcached-operator-controller-manager created
  • 查看部署结果
    [root@master memcached-operator]# kubectl get deploy -n memcached-operator-system
NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
memcached-operator-controller-manager   1/1     1            1           37m

[root@master memcached-operator]# kubectl get pods -n memcached-operator-system
NAME                                                     READY   STATUS    RESTARTS   AGE
memcached-operator-controller-manager-6fd6c7699b-42rg4   2/2     Running   0          38m
memcached-operator:Kubernetes Memcached操作员教程
03-29
memcached-operator:Kubernetes Memcached操作员教程
kubebuilder示例-新
ltgsoldier1的博客
07-07 627
SetupWithManager方法 注册要监听的资源对象Owns 监听的资源 并与控制器所管理的主资源进行关联 这样就可以在删除主资源时同时删除关联资源。因为kubebuilder 命令工具需要依赖linux环境 所以本例方法使用git 同步代码。在internal\controller\app_controller.go中写对应逻辑代码。创建api 也就是生成自定义资源相关代码 包括crd 跟controller。初始化项目 生成基础框架 domain 生成group用的。创建go mod 项目。
kubebuilder 实战之开发一个存储用户信息的 operator
qq_18681465的专栏
04-10 1215
本文介绍如何使用 kubebuilder 实现一个存储用户信息的 CRD，同时开发 controller 绑定同名的 ServiceAccount。 不过多介绍 kubebuilder 的理论知识，直接开干。 开发环境准备 初始化 kubebuilder mkdir lanyulei cd lanyulei kubebuilder init --domain lanyulei.com --repo lanyulei init：初始化命令参数。 --domain：可以理解为接口组的分组。根据实际情况来定，我一
kubebuilder实战之七：webhook
2401_84091502的博客
04-20 966
今天的文章可谓是积蓄了我这几年来的应聘和面试经历总结出来的经验，干货满满呀！如果你能够一直坚持看到这儿，那么首先我还是十分佩服你的毅力的。所以看完之后，还是多多行动起来吧！可以非常负责地说，如果你能够坚持把我上面列举的内容都一个不拉地看完并且全部消化为自己的知识的话，那么你就至少已经达到了中级开发工程师以上的水平，进入大厂技术这块是基本没有什么问题的了。着实压力不小。自己不成体系的自学效果低效又漫长，而且极易碰到天花板技术停滞不前！**
kubebuilder实战笔记
niwoxiangyu的博客
04-08 1192
先把常量准备好，可见每个pod使用的CPU和内存都是在此固定的，也可以改成在Spec中定义，这样就可以从外部传入了，另外这里为每个pod只分配了0.1个CPU可以酌情调整该值：const (// deployment中的APP标签名// tomcat容器的端口号// 单个POD的CPU资源申请// 单个POD的CPU资源上限// 单个POD的内存资源申请// 单个POD的内存资源上限。
Kubernetes Operator开发
niwoxiangyu的博客
04-19 2424
因此Crontroller 是个面向期望的编程模型,我们声明的这个期望对象，就是API Object(K8s Runtime Object)- config/crd 目录下是我们crd的描述文件,我们需要把自定义资源(CRD)的描述信息注册给k8s时需要的。- api/v1 目录下主要存放是我们API Object, 就是我们的Resource对象相关信息。部署 Controller为了确保我们的CRD描述是最新的. 我们重新安装下。
kubebuilder实战之打包部署
xingyunyang的博客
07-07 1529
kubebuilder写完operator后，如何打包并部署到任意k8s集群。
从Go到K8s Operator：我的云原生全栈实战学习笔记（含避坑指南）
crazybug1987的专栏
06-12 311
本文分享了从Go语言开发到Kubernetes Operator开发的云原生全栈实战经验，包含基础准备、容器化进阶、Kubernetes核心、Operator开发实战、全链路调试和持续交付等关键环节的避坑指南和最佳实践，特别适合有Go基础想转型云原生的开发者。
云原生 PG Operator 定制：高可用、备份恢复与弹性伸缩的大厂实现
qq_37805832的博客
04-24 408
<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Document</title> <style type="text/css"> .box1 p:nth-child(2) ...
DigitalOcean上用vLLM+Kubernetes构建专用AI推理服务
最新发布
掬水月在手，弄花香满衣
06-20 302
AI推理服务本质上是将大语言模型转化为低延迟、高可用的API能力，其核心在于计算资源调度、模型加载优化与服务治理的协同。vLLM作为高性能推理引擎，显著提升吞吐与显存利用率；Kubernetes则提供弹性扩缩、声明式部署与可观测性基础。二者结合在DigitalOcean基础设施上展现出独特优势：极简API、确定性交付、原生GPU支持及免厂商锁定的开放架构。该方案特别适合中小团队快速落地私有化Qwen、Llama等开源模型，支撑代码补全、知识库问答、轻量对话API等典型场景，兼顾工程效率与生产稳定性。
kubebuilder实战之二：初次体验kubebuilder
热门推荐
程序员欣宸的博客
01-25 1万+
通过helloworld级别的demo开发和部署，初次体验kubebuilder
k8s API编程：kubebuilder实战案例sidecarset
学亮编程手记
02-21 911
【kubebuilder 学习之初学】编写官方样例 Cronjob Controller
叮当猫的喵i
06-25 1197
/ +kubebuilder:object:root=true 标记这是一个根对象，这个类型直接对应于一个 Kubernetes API 资源，而不是嵌套在其他资源中的子资源。// +kubebuilder:resource:shortName=cj2 执行该类型的缩写为 cj2// +kubebuilder:subresource:status 为该类型生成个 status 子资源// 用户自定义 crd。
万物皆可operator之二，kubebuilder工具实现demo开发
sixinchao_1的博客
09-28 1032
本文将介绍使用上一文中搭建的kubebuilder环境开发一个operator demo。 demo功能介绍： 新建一个crd，只有一个字段 image，当我们的controller watch到有对应的crd资源创建后，使用该image创建对应的deployment，进而拉起对应的pod。 1、选择一个目录，创建Unit目录后，去执行下边的命令 mkdir Unit cd Unit go mod init Unit 会生成go.mod文件 2、然后执行，domain可自己定义。 kubebuilder
基于 kubebuilder 的 operators 的设计 & 开发
weixin_36532747的博客
04-16 630
前文提要 本篇开始开发一个有实际作用的operator，该operator名为elasticweb，既弹性web服务，本篇参考网络的文章进行汇总实践； 这将是一次完整的operator开发实战，设计、编码、部署等环节都会涉及到每一个产品的细节和具体实现，elasticweb从CRD设计再到controller功能都有明确的业务含义，能执行业务逻辑。 需求背景 设计架构 背景：做过网站开发的同学对横向扩容应该都了解，简单的说，假设一个tomcat的QPS上限为500，如果外部访问的QPS达到了600，为了保障
Kubebuilder：Go语言开发Kubernetes控制器全面教程
weixin_29607511的博客
08-14 1086
自定义资源定义（CRD）使得用户可以在 Kubernetes 中创建新的资源类型，扩展 API。CRD 与控制器之间的交互是通过 Kubernetes API 实现的，控制器监听 API 事件并作出响应。这些机制是使用 Kubebuilder 框架开发自定义控制器不可或缺的部分。通过以上内容，我们为深入理解 Kubebuilder 框架奠定了基础，接下来的章节将具体介绍 Kubebuilder 的设计理念和组件。在Kubebuilder中，资源定义是通过自定义资源（CR）来体现的。
体验 kubebuilder 开发 kubernetes operator
engchina的专栏
02-08 1225
体验 kubebuilder 开发 kubernetes operator
Kubernetes生产实践系列之二十七：Kubernetes Operator之构建memcached operator并进行部署
cloudvtech的博客
09-12 805
一、前言
万字长文 | 面向k8s编程，如何写一个Operator
NewTyun的博客
05-16 4936
新钛云服已为您服务1489天概述随着我们对 Kubernetes 的逐步了解，可能就会发现 Kubernetes 中内置的对象定义，比如 Deployment、StatefulSet、ConfigMap，可能已经不能满足我们的需求。我们希望在 Kubernetes 定义一些自己的对象，一是可以通过 kube-apiserver 提供统一的访问入口，二是可以像其他内置对象一...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值