Press "Enter" to skip to content

在 kubernetes部署和使用 jenkins（二）

Published 2023年7月11日 by chenjie
内容目录

上文讲了在 kubernetes 内部署 jenkins 和执行任务，但是如果执行多个关联任务也就是我们常常说的流水线就需要使用到插件 pipeline的能力，本文讲用一个示例进行说明。

pipeline

准备

在harbor(部署可参考https://chenjie.info/2664) ，创建chenjie 这个项目

gitlab（部署可参考https://chenjie.info/2729)

安装插件

git

Untitled

Pipeline Stage View

Untitled

coredns配置

增加域名解析方便对gitlab harbor的访问

kubectl edit cm coredns -n kube-system

Untitled

如果apiserver 配置了域名也需加入方便后续有在slave pod 中kubectl 等操作

如果集群使用了nodelocaldns

kubectl edit cm -n kube-system nodelocaldns

需要修改.:53中forward 和上面区块中的保持一致，以保证集群pod DNS解析都会走coredns

Untitled

执行流水线

创建流水线任务devops-demo

Untitled

Untitled

这里的secret token在gitlab的项目中webhook中https://gitlab.chenjie.info/chenjie/devops-demo/-/hooks/1/edit配置

https://jenkins.chenjie.info/project/devops-demo

Untitled

在gitlab使用root账户确保webhook可访问外部

https://gitlab.chenjie.info/admin/application_settings/network

Untitled

在webhook中https://gitlab.chenjie.info/chenjie/devops-demo/-/hooks/1/edit

可以点击test 确保返回码200

Untitled

Untitled

创建流水线中所需的凭证

https://jenkins.chenjie.info/manage/credentials/

Untitled

Untitled

上传kubeconfig

Untitled

https://gitlab.chenjie.info/chenjie/devops-demo/-/blob/main/Jenkinsfile git项目源代码https://github.com/jaychenthinkfast/devops-demo 需要克隆到本地gitlab 按需修改Jenkinsfile

def label = "slave-${UUID.randomUUID().toString()}"

def helmLint(String chartDir) {
    println "校验 chart 模板"
    sh "helm lint ${chartDir}"
}

def helmDeploy(Map args) {
    if (args.debug) {
        println "Debug 应用"
        sh "helm upgrade --dry-run --debug --install ${args.name} ${args.chartDir} -f ${args.valuePath} --set image.tag=${args.imageTag} --namespace ${args.namespace}"
    } else {
        println "部署应用"
        sh "helm upgrade --install ${args.name} ${args.chartDir} -f ${args.valuePath} --set image.tag=${args.imageTag} --namespace ${args.namespace}"
        echo "应用 ${args.name} 部署成功. 可以使用 helm status ${args.name} 查看应用状态"
    }
}

podTemplate(label: label, containers: [
  containerTemplate(name: 'golang', image: 'golang:1.14.2-alpine3.11', command: 'cat', ttyEnabled: true),
  containerTemplate(name: 'docker', image: 'docker:latest', command: 'cat', ttyEnabled: true),
  containerTemplate(name: 'helm', image: 'cnych/helm', command: 'cat', ttyEnabled: true),
  containerTemplate(name: 'kubectl', image: 'cnych/kubectl', command: 'cat', ttyEnabled: true)
], serviceAccount: 'jenkins', envVars: [
  envVar(key: 'DOCKER_HOST', value: 'tcp://docker-dind:2375'),
  envVar(key: 'GIT_SSL_NO_VERIFY', value: 'true') 
]) {
  node(label) {
    def myRepo = checkout scm
    // 获取 git commit id 作为镜像标签
    def imageTag = sh(script: "git rev-parse --short HEAD", returnStdout: true).trim()
    // 仓库地址
    def registryUrl = "harbor.chenjie.info"
    def imageEndpoint = "chenjie/devops-demo"
    // 镜像
    def image = "${registryUrl}/${imageEndpoint}:${imageTag}"

    stage('单元测试') {
      echo "测试阶段"
    }
    stage('代码编译打包') {
      try {
        container('golang') {
          echo "2.代码编译打包阶段"
          sh """
            export GOPROXY=https://goproxy.cn
            GOOS=linux GOARCH=amd64 go build -v -o demo-app
            """
        }
      } catch (exc) {
        println "构建失败 - ${currentBuild.fullDisplayName}"
        throw(exc)
      }
    }
    stage('构建 Docker 镜像') {
      withCredentials([[$class: 'UsernamePasswordMultiBinding',
        credentialsId: 'docker-auth',
        usernameVariable: 'DOCKER_USER',
        passwordVariable: 'DOCKER_PASSWORD']]) {
          container('docker') {
            echo "3. 构建 Docker 镜像阶段"
            sh """
              cat /etc/resolv.conf
              docker login ${registryUrl} -u ${DOCKER_USER} -p ${DOCKER_PASSWORD}
              docker build -t ${image} .
              docker push ${image}
              """
          }
      }
    }
    stage('运行 Helm') {
      withCredentials([file(credentialsId: 'kubeconfig', variable: 'KUBECONFIG')]) {
        container('helm') {
          sh "mkdir -p ~/.kube && cp ${KUBECONFIG} ~/.kube/config"
          echo "4.开始 Helm 部署"
          def userInput = input(
            id: 'userInput',
            message: '选择一个部署环境',
            parameters: [
                [
                    $class: 'ChoiceParameterDefinition',
                    choices: "Dev\nQA\nProd",
                    name: 'Env'
                ]
            ]
          )
          echo "部署应用到 ${userInput} 环境"
          // 选择不同环境下面的 values 文件
          if (userInput == "Dev") {
              // deploy dev stuff
          } else if (userInput == "QA"){
              // deploy qa stuff
          } else {
              // deploy prod stuff
          }
          helmDeploy(
              debug       : false,
              name        : "devops-demo",
              chartDir    : "./helm",
              namespace   : "kube-ops",
              valuePath   : "./helm/my-values.yaml",
              imageTag    : "${imageTag}"
          )
        }
      }
    }
    stage('运行 Kubectl') {
      withCredentials([file(credentialsId: 'kubeconfig', variable: 'KUBECONFIG')]) {
        container('kubectl') {
          sh "mkdir -p ~/.kube && cp ${KUBECONFIG} ~/.kube/config"
          echo "5.查看应用"
          sh "kubectl get all -n kube-ops -l app=devops-demo"
        }
      }
    }
    stage('快速回滚?') {
      withCredentials([file(credentialsId: 'kubeconfig', variable: 'KUBECONFIG')]) {
        container('helm') {
          sh "mkdir -p ~/.kube && cp ${KUBECONFIG} ~/.kube/config"
          def userInput = input(
            id: 'userInput',
            message: '是否需要快速回滚？',
            parameters: [
                [
                    $class: 'ChoiceParameterDefinition',
                    choices: "Y\nN",
                    name: '回滚?'
                ]
            ]
          )
          if (userInput == "Y") {
            sh "helm rollback devops-demo --namespace kube-ops"
          }
        }
      }
    }
  }
}

按需修改 helm/my-values.yaml

image:
  repository: harbor.chenjie.info/chenjie/devops-demo
  tag: latest
  pullPolicy: IfNotPresent

ingress:
  enabled: true
  ingressClassName: traefik
  path: /
  hosts:
    - devops-demo.chenjie.info

resources:
  limits:
    cpu: 50m
    memory: 128Mi
  requests:
    cpu: 50m
    memory: 128Mi

执行视图

Untitled

控制台输出

Started by GitLab push by jie chen
Obtained Jenkinsfile from git https://gitlab.chenjie.info/chenjie/devops-demo.git
[Pipeline] Start of Pipeline
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Created Pod: kubernetes kube-ops/slave-b4aa2b40-c1fd-46d0-a380-9063044f3c0a-190t5-czqw2
Still waiting to schedule task
Waiting for next available executor on ‘slave-b4aa2b40-c1fd-46d0-a380-9063044f3c0a-190t5-czqw2’
Agent slave-b4aa2b40-c1fd-46d0-a380-9063044f3c0a-190t5-czqw2 is provisioned from template slave-b4aa2b40-c1fd-46d0-a380-9063044f3c0a-190t5
---
apiVersion: "v1"
kind: "Pod"
metadata:
  annotations:
    buildUrl: "http://jenkins.kube-ops.svc.cluster.local:8080/job/devops-demo/13/"
    runUrl: "job/devops-demo/13/"
  labels:
    jenkins: "slave"
    jenkins/label-digest: "cfc390563aba22bb76c772551ce54f921748bba3"
    jenkins/label: "slave-b4aa2b40-c1fd-46d0-a380-9063044f3c0a"
  name: "slave-b4aa2b40-c1fd-46d0-a380-9063044f3c0a-190t5-czqw2"
  namespace: "kube-ops"
spec:
  containers:
  - command:
    - "cat"
    env:
    - name: "DOCKER_HOST"
      value: "tcp://docker-dind:2375"
    - name: "GIT_SSL_NO_VERIFY"
      value: "true"
    image: "golang:1.14.2-alpine3.11"
    imagePullPolicy: "IfNotPresent"
    name: "golang"
    resources: {}
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
      readOnly: false
  - command:
    - "cat"
    env:
    - name: "DOCKER_HOST"
      value: "tcp://docker-dind:2375"
    - name: "GIT_SSL_NO_VERIFY"
      value: "true"
    image: "cnych/kubectl"
    imagePullPolicy: "IfNotPresent"
    name: "kubectl"
    resources: {}
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
      readOnly: false
  - command:
    - "cat"
    env:
    - name: "DOCKER_HOST"
      value: "tcp://docker-dind:2375"
    - name: "GIT_SSL_NO_VERIFY"
      value: "true"
    image: "cnych/helm"
    imagePullPolicy: "IfNotPresent"
    name: "helm"
    resources: {}
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
      readOnly: false
  - command:
    - "cat"
    env:
    - name: "DOCKER_HOST"
      value: "tcp://docker-dind:2375"
    - name: "GIT_SSL_NO_VERIFY"
      value: "true"
    image: "docker:latest"
    imagePullPolicy: "IfNotPresent"
    name: "docker"
    resources: {}
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
      readOnly: false
  - env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "jenkins.kube-ops.svc.cluster.local:50000"
    - name: "GIT_SSL_NO_VERIFY"
      value: "true"
    - name: "JENKINS_AGENT_NAME"
      value: "slave-b4aa2b40-c1fd-46d0-a380-9063044f3c0a-190t5-czqw2"
    - name: "DOCKER_HOST"
      value: "tcp://docker-dind:2375"
    - name: "JENKINS_NAME"
      value: "slave-b4aa2b40-c1fd-46d0-a380-9063044f3c0a-190t5-czqw2"
    - name: "JENKINS_AGENT_WORKDIR"
      value: "/home/jenkins/agent"
    - name: "JENKINS_URL"
      value: "http://jenkins.kube-ops.svc.cluster.local:8080/"
    image: "jenkins/inbound-agent:3107.v665000b_51092-5"
    name: "jnlp"
    resources:
      requests:
        memory: "256Mi"
        cpu: "100m"
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
      readOnly: false
  nodeSelector:
    kubernetes.io/os: "linux"
  restartPolicy: "Never"
  serviceAccountName: "jenkins"
  volumes:
  - emptyDir:
      medium: ""
    name: "workspace-volume"

Running on slave-b4aa2b40-c1fd-46d0-a380-9063044f3c0a-190t5-czqw2 in /home/jenkins/agent/workspace/devops-demo
[Pipeline] {
[Pipeline] checkout
Selected Git installation does not exist. Using Default
The recommended git tool is: NONE
using credential gitlab-auth
Cloning the remote Git repository
Avoid second fetch
skipping resolution of commit remotes/origin/main, since it originates from another repository
Checking out Revision d51ddaf8850b77223936a1bfa5bc8c924230a61e (origin/main)
Cloning repository https://gitlab.chenjie.info/chenjie/devops-demo.git
 > git init /home/jenkins/agent/workspace/devops-demo # timeout=10
Fetching upstream changes from https://gitlab.chenjie.info/chenjie/devops-demo.git
 > git --version # timeout=10
 > git --version # 'git version 2.30.2'
using GIT_ASKPASS to set credentials gitlab认证信息
 > git fetch --tags --force --progress -- https://gitlab.chenjie.info/chenjie/devops-demo.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://gitlab.chenjie.info/chenjie/devops-demo.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git rev-parse origin/main^{commit} # timeout=10
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d51ddaf8850b77223936a1bfa5bc8c924230a61e # timeout=10
Commit message: "Merge branch 'main-patch-e6a1' into 'main'"
 > git rev-list --no-walk 73cbe663553ee0782bc122466f4aa90299ee92d4 # timeout=10
[Pipeline] sh
+ git rev-parse --short HEAD
[Pipeline] stage
[Pipeline] { (单元测试)
[Pipeline] echo
测试阶段
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (代码编译打包)
[Pipeline] container
[Pipeline] {
[Pipeline] echo
2.代码编译打包阶段
[Pipeline] sh
+ export 'GOPROXY=https://goproxy.cn'
+ GOOS=linux GOARCH=amd64 go build -v -o demo-app
go: downloading github.com/sirupsen/logrus v1.4.2
go: downloading github.com/gin-gonic/gin v1.4.0
go: downloading golang.org/x/sys v0.0.0-20190422165155-953cdadca894
go: downloading github.com/gin-contrib/sse v0.0.0-20190301062529-5545eab6dad3
go: downloading github.com/mattn/go-isatty v0.0.7
go: downloading gopkg.in/go-playground/validator.v8 v8.18.2
go: downloading github.com/golang/protobuf v1.3.1
go: downloading gopkg.in/yaml.v2 v2.2.2
go: downloading github.com/ugorji/go v1.1.4
github.com/gin-gonic/gin/internal/json
github.com/golang/protobuf/proto
gopkg.in/go-playground/validator.v8
gopkg.in/yaml.v2
golang.org/x/sys/unix
github.com/gin-contrib/sse
github.com/ugorji/go/codec
github.com/mattn/go-isatty
github.com/sirupsen/logrus
github.com/gin-gonic/gin/binding
github.com/gin-gonic/gin/render
github.com/gin-gonic/gin
dronek8s
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (构建 Docker 镜像)
[Pipeline] withCredentials
Masking supported pattern matches of $DOCKER_PASSWORD
[Pipeline] {
[Pipeline] container
[Pipeline] {
[Pipeline] echo
3. 构建 Docker 镜像阶段
[Pipeline] sh
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
         Affected argument(s) used the following variable(s): [DOCKER_PASSWORD]
         See https://jenkins.io/redirect/groovy-string-interpolation for details.
+ cat /etc/resolv.conf
nameserver 169.254.25.10
search kube-ops.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
+ docker login harbor.chenjie.info -u admin -p ****
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
+ docker build -t harbor.chenjie.info/chenjie/devops-demo:d51ddaf .
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 445B done
#2 DONE 0.0s

#3 [internal] load metadata for dockerproxy.com/library/alpine:3.16.2
#3 DONE 1.6s

#4 [1/4] FROM dockerproxy.com/library/alpine:3.16.2@sha256:65a2763f593ae85fab3b5406dc9e80f744ec5b449f269b699b5efd37a07ad32e
#4 DONE 0.0s

#5 [internal] load build context
#5 transferring context: 16.94MB 0.1s done
#5 DONE 0.1s

#6 [2/4] WORKDIR /home
#6 CACHED

#7 [3/4] RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories &&   apk update &&   apk upgrade &&   apk add ca-certificates && update-ca-certificates &&   apk add --update tzdata &&   rm -rf /var/cache/apk/*
#7 CACHED

#8 [4/4] COPY demo-app /home/
#8 CACHED

#9 exporting to image
#9 exporting layers done
#9 writing image sha256:f2006adc0c3229418b34f903bfe94e3ae2d43e6aac944d89f1e872a7e63bba44 done
#9 naming to harbor.chenjie.info/chenjie/devops-demo:d51ddaf done
#9 DONE 0.0s
WARNING: buildx: git was not found in the system. Current commit information was not captured by the build
+ docker push harbor.chenjie.info/chenjie/devops-demo:d51ddaf
The push refers to repository [harbor.chenjie.info/chenjie/devops-demo]
024c0e2103c3: Preparing
52f64311adb3: Preparing
5f70bf18a086: Preparing
994393dc58e7: Preparing
5f70bf18a086: Layer already exists
52f64311adb3: Layer already exists
994393dc58e7: Layer already exists
024c0e2103c3: Layer already exists
d51ddaf: digest: sha256:45e19b9e01c42494e51156f979d4ccabb48e4e6cdecda82cf098bfd9020a83e2 size: 1156
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (运行 Helm)
[Pipeline] withCredentials
Masking supported pattern matches of $KUBECONFIG
[Pipeline] {
[Pipeline] container
[Pipeline] {
[Pipeline] sh
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
         Affected argument(s) used the following variable(s): [KUBECONFIG]
         See https://jenkins.io/redirect/groovy-string-interpolation for details.
+ mkdir -p /root/.kube
+ cp **** /root/.kube/config
[Pipeline] echo
4.开始 Helm 部署
[Pipeline] input
Input requested
Approved by admin
[Pipeline] echo
部署应用到 Dev 环境
[Pipeline] echo
部署应用
[Pipeline] sh
+ helm upgrade --install devops-demo ./helm -f ./helm/my-values.yaml --set 'image.tag=d51ddaf' --namespace kube-ops
Release "devops-demo" has been upgraded. Happy Helming!
NAME: devops-demo
LAST DEPLOYED: Fri Apr 28 12:37:06 2023
NAMESPACE: kube-ops
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  http://devops-demo.chenjie.info/
[Pipeline] echo
应用 devops-demo 部署成功. 可以使用 helm status devops-demo 查看应用状态
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (运行 Kubectl)
[Pipeline] withCredentials
Masking supported pattern matches of $KUBECONFIG
[Pipeline] {
[Pipeline] container
[Pipeline] {
[Pipeline] sh
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
         Affected argument(s) used the following variable(s): [KUBECONFIG]
         See https://jenkins.io/redirect/groovy-string-interpolation for details.
+ mkdir -p /root/.kube
+ cp **** /root/.kube/config
[Pipeline] echo
5.查看应用
[Pipeline] sh
+ kubectl get all -n kube-ops -l 'app=devops-demo'
NAME                               READY   STATUS              RESTARTS   AGE
pod/devops-demo-7fcdf94969-8w6c7   0/1     ContainerCreating   0          1s

NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/devops-demo   ClusterIP   10.233.24.229           80/TCP    3h40m

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/devops-demo   0/1     1            0           3h40m

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/devops-demo-7fcdf94969   1         1         0       1
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (快速回滚?)
[Pipeline] withCredentials
Masking supported pattern matches of $KUBECONFIG
[Pipeline] {
[Pipeline] container
[Pipeline] {
[Pipeline] sh
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
         Affected argument(s) used the following variable(s): [KUBECONFIG]
         See https://jenkins.io/redirect/groovy-string-interpolation for details.
+ mkdir -p /root/.kube
+ cp **** /root/.kube/config
[Pipeline] input
Input requested
Approved by admin
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS

解析devops-demo.chenjie.info 到traefik 所在节点ip 即可访问该服务

curl http://devops-demo.chenjie.info

{"msg":"this is pipeline test by chenjie.info"}

参考

  1. https://www.qikqiak.com/post/resolve-coredns-hosts-invalid/
  2. https://docs.youdianzhishi.com/k8s/devops/pipeline/

Published in kubernetes

发表回复

您的邮箱地址不会被公开。 必填项已用 * 标注