Press "Enter" to skip to content

使用kubebuilder构建 memcached operator

Published 2024年1月16日 by chenjie
内容目录

需求

这是根据openshift 文档改写的一个示例，openshift使用Operator SDK来构建memcached operator，这里我们用 kubebuilder 进行重写。

crd 的内容很简单，spec部分主要是 memcached 的数量，主要用于修正其 pod 数量，

spec:
  size: 1

而在 status部分展示了 memcached 的 pod 名slice。

status:
  nodes:
  - memcached-sample-5c5b999498-q9x7l

工具

这里需要用到 kubebuilder、go，需要注意版本问题，例如我这里用的 kubernetes 1.27.x版本，那么对应的 kubebuilder 应该使用3.11.x, go 版本为 1.20.x。具体对应版本可以查看各个工具的 github 官方说明。

operator 开发流程

初始化项目并指定域名

kubebuilder init --domain chenjie.info

输出

Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.15.0
Update dependencies:
$ go mod tidy
Next: define a resource with:
$ kubebuilder create api

这里主要是生成一些 manifests 模板以供后续使用和更新依赖，根据提示下一步应该进行 api 的创建

创建 api

kubebuilder create api --group=cache --version=v1alpha1 --kind=Memcached

提示输入

Create Resource [y/n]
y
Create Controller [y/n]
y

这里需要生成 api 资源和创建控制器，所以都选择 y

输出

Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
api/v1alpha1/memcached_types.go
api/v1alpha1/groupversion_info.go
internal/controller/suite_test.go
internal/controller/memcached_controller.go
Update dependencies:
$ go mod tidy
Running make:
$ make generate
mkdir -p /Users/chenjie/work/go/src/kube/memcached-operator1/bin
test -s /Users/chenjie/work/go/src/kube/memcached-operator1/bin/controller-gen && /Users/chenjie/work/go/src/kube/memcached-operator1/bin/controller-gen --version | grep -q v0.12.0 || \
        GOBIN=/Users/chenjie/work/go/src/kube/memcached-operator1/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.12.0
/Users/chenjie/work/go/src/kube/memcached-operator1/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests

这里生成了 api 和控制器模板，按照提示下一步需要修改 api 逻辑并生成 manifests 用于创建 crd到集群中。

修改 api

根据前面的需求逻辑

这里主要对 spec 和 status 进行修改, 涉及改动如下

// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required.  Any new fields you add must have json tags for the fields to be serialized.

// MemcachedSpec defines the desired state of Memcached
type MemcachedSpec struct {
    // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
    // Important: Run "make" to regenerate code after modifying this file

    // +kubebuilder:validation:Minimum=0
    // Size is the size of the memcached deployment
    Size int32 json:"size"
}

// MemcachedStatus defines the observed state of Memcached
type MemcachedStatus struct {
    // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
    // Important: Run "make" to regenerate code after modifying this file

    // Nodes are the names of the memcached pods
    Nodes []string json:"nodes"
}

执行

make manifests

输出

/Users/chenjie/work/go/src/kube/memcached-operator1/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases

实现 api 逻辑后生成 manifests。

生成深拷贝代码：

make generate

安装 api

这里就把 api 所生成的 crd 安装到集群中，如果是本地集群直接执行下面命令，如果远程集群可以配置 kubeconfig 到本地再执行或者拷贝config/crd下内容到目标集群进行执行（/Users/chenjie/work/go/src/kube/memcached-operator1/bin/kustomize build config/crd | kubectl apply -f -）

make install

输出

/Users/chenjie/work/go/src/kube/memcached-operator1/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
test -s /Users/chenjie/work/go/src/kube/memcached-operator1/bin/kustomize || GOBIN=/Users/chenjie/work/go/src/kube/memcached-operator1/bin GO111MODULE=on go install sigs.k8s.io/kustomize/kustomize/v5@v5.0.1

customresourcedefinition.apiextensions.k8s.io/memcacheds.cache.chenjie.info created

实现控制器逻辑

/*
Copyright 2024.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package controller

import (
    "context"
    "github.com/go-logr/logr"
    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/api/errors"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    "reflect"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/log"

    cachev1alpha1 "kube/memcached-operator1/api/v1alpha1"
)

// MemcachedReconciler reconciles a Memcached object
type MemcachedReconciler struct {
    client.Client
    Log    logr.Logger
    Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=cache.chenjie.info,resources=memcacheds,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=cache.chenjie.info,resources=memcacheds/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=cache.chenjie.info,resources=memcacheds/finalizers,verbs=update
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;watch;

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Memcached object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.15.0/pkg/reconcile
func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    _ = log.FromContext(ctx)

    // TODO(user): your logic here
    log := r.Log.WithValues("memcached", req.NamespacedName)

    // Fetch the Memcached instance
    memcached := &cachev1alpha1.Memcached{}
    err := r.Get(ctx, req.NamespacedName, memcached)
    if err != nil {
        if errors.IsNotFound(err) {
            // Request object not found, could have been deleted after reconcile request.
            // Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
            // Return and don't requeue
            log.Info("Memcached resource not found. Ignoring since object must be deleted")
            return ctrl.Result{}, nil
        }
        // Error reading the object - requeue the request.
        log.Error(err, "Failed to get Memcached")
        return ctrl.Result{}, err
    }

    // Check if the deployment already exists, if not create a new one
    found := &appsv1.Deployment{}
    err = r.Get(ctx, req.NamespacedName, found)
    if err != nil && errors.IsNotFound(err) {
        // Define a new deployment
        dep := r.deploymentForMemcached(memcached)
        log.Info("Creating a new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
        err = r.Create(ctx, dep)
        if err != nil {
            log.Error(err, "Failed to create new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
            return ctrl.Result{}, err
        }
        // Deployment created successfully - return and requeue
        return ctrl.Result{Requeue: true}, nil
    } else if err != nil {
        log.Error(err, "Failed to get Deployment")
        return ctrl.Result{}, err
    }

    // Ensure the deployment size is the same as the spec
    size := memcached.Spec.Size
    newFound := found.DeepCopy()
    if *newFound.Spec.Replicas != size {
        newFound.Spec.Replicas = &size
        err = r.Patch(ctx, newFound, client.MergeFrom(found))
        if err != nil {
            log.Error(err, "Failed to update Deployment", "Deployment.Namespace", found.Namespace, "Deployment.Name", found.Name)
            return ctrl.Result{}, err
        }
        return ctrl.Result{Requeue: true}, nil
    }

    // Update the Memcached status with the pod names
    // List the pods for this memcached's deployment
    podList := &corev1.PodList{}
    listOpts := []client.ListOption{
        client.InNamespace(memcached.Namespace),
        client.MatchingLabels(labelsForMemcached(memcached.Name)),
    }
    if err = r.List(ctx, podList, listOpts...); err != nil {
        log.Error(err, "Failed to list pods", "Memcached.Namespace", memcached.Namespace, "Memcached.Name", memcached.Name)
        return ctrl.Result{}, err
    }
    podNames := getPodNames(podList.Items)
    if int32(len(podNames)) != size {
        return ctrl.Result{Requeue: true}, nil
    }
    // Update status.Nodes if needed
    if !reflect.DeepEqual(podNames, memcached.Status.Nodes) {
        newMemcached := memcached.DeepCopy()
        newMemcached.Status.Nodes = podNames
        err := r.Status().Patch(ctx, newMemcached, client.MergeFrom(memcached))
        if err != nil {
            log.Error(err, "Failed to update Memcached status")
            return ctrl.Result{}, err
        }
    }

    return ctrl.Result{}, nil
}

// deploymentForMemcached returns a memcached Deployment object
func (r *MemcachedReconciler) deploymentForMemcached(m *cachev1alpha1.Memcached) *appsv1.Deployment {
    ls := labelsForMemcached(m.Name)
    replicas := m.Spec.Size

    dep := &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      m.Name,
            Namespace: m.Namespace,
        },
        Spec: appsv1.DeploymentSpec{
            Replicas: &replicas,
            Selector: &metav1.LabelSelector{
                MatchLabels: ls,
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: ls,
                },
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{{
                        Image:   "memcached:1.4.36-alpine",
                        Name:    "memcached",
                        Command: []string{"memcached", "-m=64", "-o", "modern", "-v"},
                        Ports: []corev1.ContainerPort{{
                            ContainerPort: 11211,
                            Name:          "memcached",
                        }},
                    }},
                },
            },
        },
    }
    // Set Memcached instance as the owner and controller
    ctrl.SetControllerReference(m, dep, r.Scheme)
    return dep
}

// labelsForMemcached returns the labels for selecting the resources
// belonging to the given memcached CR name.
func labelsForMemcached(name string) map[string]string {
    return map[string]string{"app": "memcached", "memcached_cr": name}
}

// getPodNames returns the pod names of the array of pods passed in
func getPodNames(pods []corev1.Pod) []string {
    var podNames []string
    for _, pod := range pods {
        podNames = append(podNames, pod.Name)
    }
    return podNames
}

// SetupWithManager sets up the controller with the Manager.
func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&cachev1alpha1.Memcached{}).
        Owns(&appsv1.Deployment{}).
        Complete(r)
}

这里代码逻辑主要是根据 crd对象中的 size 对 memcached 的 deployment 的 size进行调谐，让最终的 memcached 的 pod 数量总是和 size 是一致的。

这里特别需要注意的子资源对应的监听管理

// SetupWithManager sets up the controller with the Manager.
func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&cachev1alpha1.Memcached{}).
        Owns(&appsv1.Deployment{}).   //监听 memcached 的 deployment资源
        Complete(r)
}

还有就是子资源的权限控制，主要在调谐控制器的注解中体现

// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;watch;

控制器本地测试

在打镜像部署到集群之前建议在本地先进行命令行直接测试，命令：

make run

输出

test -s /Users/chenjie/work/go/src/kube/memcached-operator1/bin/controller-gen && /Users/chenjie/work/go/src/kube/memcached-operator1/bin/controller-gen --version | grep -q v0.12.0 || \
        GOBIN=/Users/chenjie/work/go/src/kube/memcached-operator1/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.12.0
/Users/chenjie/work/go/src/kube/memcached-operator1/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/Users/chenjie/work/go/src/kube/memcached-operator1/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go run ./cmd/main.go

修改config/samples/cache_v1alpha1_memcached.yaml

apiVersion: cache.chenjie.info/v1alpha1
kind: Memcached
metadata:
  labels:
    app.kubernetes.io/name: memcached
    app.kubernetes.io/instance: memcached-sample
    app.kubernetes.io/part-of: memcached-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: memcached-operator
  name: memcached-sample
spec:
  size: 1

apply 到集群中

kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml

查看 crd对象 status ，主要看 status.Nodes 数量是否和 size一致

kubectl describe memcacheds.cache.chenjie.info memcached-sample 
输出：  
Name:         memcached-sample
Namespace:    default
Labels:       app.kubernetes.io/created-by=memcached-operator
              app.kubernetes.io/instance=memcached-sample
              app.kubernetes.io/managed-by=kustomize
              app.kubernetes.io/name=memcached
              app.kubernetes.io/part-of=memcached-operator
Annotations:  
API Version:  cache.chenjie.info/v1alpha1
Kind:         Memcached
Metadata:
  Creation Timestamp:  2024-01-16T06:03:13Z
  Generation:          1
  Resource Version:    6016537
  UID:                 486069bf-b780-495b-b11e-73fac8bac851
Spec:
  Size:  1
Status:
  Nodes:
    memcached-sample-5c5b999498-q9x7l
Events:

修改 size 到 3再次更新对象

再次检查，验证通过

kube kubectl describe memcacheds.cache.chenjie.info memcached-sample
输出
Name:         memcached-sample
Namespace:    default
Labels:       app.kubernetes.io/created-by=memcached-operator
              app.kubernetes.io/instance=memcached-sample
              app.kubernetes.io/managed-by=kustomize
              app.kubernetes.io/name=memcached
              app.kubernetes.io/part-of=memcached-operator
Annotations:  
API Version:  cache.chenjie.info/v1alpha1
Kind:         Memcached
Metadata:
  Creation Timestamp:  2024-01-16T06:03:13Z
  Generation:          2
  Resource Version:    6016752
  UID:                 486069bf-b780-495b-b11e-73fac8bac851
Spec:
  Size:  3
Status:
  Nodes:
    memcached-sample-5c5b999498-q9x7l
    memcached-sample-5c5b999498-qpxzj
    memcached-sample-5c5b999498-l6bkw
Events:

修改 Dockerfile

本地测试后需要进行镜像制作，建议增加构建镜像中 goproxy 配置，和运行镜像中的 gcr 镜像替换国内镜像操作。

# Build the manager binary
FROM golang:1.20 as builder

WORKDIR /workspace
# Copy the Go Modules manifests
COPY go.mod go.mod
COPY go.sum go.sum
ENV GOPROXY=https://goproxy.cn
# cache deps before building and copying source so that we don't need to re-download as much
# and so that source changes don't invalidate our downloaded layer
RUN go mod download

# Copy the go source
COPY cmd/main.go cmd/main.go
COPY api/ api/
COPY internal/controller/ internal/controller/

# Build
# the GOARCH has not a default value to allow the binary be built according to the host where the command
# was called. For example, if we call make docker-build in a local env which has the Apple Silicon M1 SO
# the docker BUILDPLATFORM arg will be linux/arm64 when for Apple x86 it will be linux/amd64. Therefore,
# by leaving it empty we can ensure that the container and binary shipped on it will have the same platform.
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager cmd/main.go

# Use distroless as minimal base image to package the manager binary
# Refer to https://github.com/GoogleContainerTools/distroless for more details
FROM gcr.m.daocloud.io/distroless/static:nonroot
WORKDIR /
COPY --from=builder /workspace/manager .
USER 65532:65532

ENTRYPOINT ["/manager"]

修改Makefile

修改镜像参数，也可以在命名行执行代入

IMG ?= chenjie.info/operator/memcached:v0.0.2

镜像构建

make docker-build

镜像推送

如何推送私有仓库，需要在本地配置镜像证书并先行登录验证后推送

make docker-push

部署控制器

这里会替换控制器manifests 中 img的内容

make deploy

卸载控制器

make undeploy

卸载 api

make uninstall

源码

https://github.com/jaychenthinkfast/memcached-operator

参考

  1. https://docs.openshift.com/container-platform/4.15/operators/operator_sdk/golang/osdk-golang-tutorial.html
  2. https://github.com/kubernetes-sigs/kubebuilder

Published in kubernetes

发表回复

您的电子邮箱地址不会被公开。 必填项已用 * 标注