在 kubernetes部署和使用 jenkins(一)

前置说明

本文档基于kubernetes 1.22版本使用,其他版本可能存在版本兼容问题。

前置准备

rancher local-path部署

为Jenkins提供动态pv

kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.24/deploy/local-path-storage.yaml

部署后

kubectl get sc

NAME                    PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path              rancher.io/local-path   Delete          WaitForFirstConsumer   false                  22d

Traefik安装和域名配置

为Jenkins提供域名入口支持

https://chenjie.info/2698

Jenkins部署

部署资源清单jenkins.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pvc
  namespace: kube-ops
spec:
  storageClassName: local-path  # 指定一个可用的 storageclass
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: kube-ops
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jenkins
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments", "ingresses"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
  - apiGroups: [""]
    resources: ["pods/log", "events"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins
  namespace: kube-ops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins
subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: kube-ops
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: kube-ops
spec:
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      serviceAccount: jenkins
      initContainers:
        - name: fix-permissions
          image: busybox:1.35.0
          command: ["sh", "-c", "chown -R 1000:1000 /var/jenkins_home"]
          securityContext:
            privileged: true
          volumeMounts:
            - name: jenkinshome
              mountPath: /var/jenkins_home
      containers:
        - name: jenkins
          image: jenkins/jenkins:2.402
          imagePullPolicy: IfNotPresent
          env:
          - name: JAVA_OPTS
            value: -Dhudson.model.DownloadService.noSignatureCheck=true
          - name: GIT_SSL_NO_VERIFY #拉取代码时跳过ssl认证
            value: "true"
          ports:
            - containerPort: 8080
              name: web
              protocol: TCP
            - containerPort: 50000
              name: agent
              protocol: TCP
          resources:
            limits:
              cpu: 1500m
              memory: 2048Mi
            requests:
              cpu: 1500m
              memory: 2048Mi
          readinessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12
          volumeMounts:
            - name: jenkinshome
              mountPath: /var/jenkins_home
      volumes:
        - name: jenkinshome
          persistentVolumeClaim:
            claimName: jenkins-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: kube-ops
  labels:
    app: jenkins
spec:
  selector:
    app: jenkins
  ports:
    - name: web
      port: 8080
      targetPort: web
    - name: agent
      port: 50000
      targetPort: agent
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: jenkins
  namespace: kube-ops
spec:
  entryPoints:
  - websecure
  routes:
  - match: Host(jenkins.chenjie.info)  # 指定域名
    kind: Rule
    services:
    - name: jenkins
      port: 8080
  tls:
    secretName: jenkins-ssl

上面清单中使用了https协议作为入口访问,需要用到tls,这里使用自签名证书为例

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=jenkins.chenjie.info"
kubectl create ns kube-ops
kubectl create secret tls jenkins-ssl -n kube-ops --cert=tls.crt --key=tls.key

执行Jenkins部署指令

kubectl apply -f jenkins.yaml

访问Jenkins web 服务

部署后在本地hosts 映射域名地址,访问https://jenkins.chenjie.info

密码获取方式如下

kubectl get pods -n kube-ops 

NAME                       READY   STATUS    RESTARTS      AGE
jenkins-7c69677d77-49rp2   1/1     Running   3 (15h ago)   15h

kubectl exec -it jenkins-7c69677d77-49rp2 -n kube-ops -- cat /var/jenkins_home/secrets/initialAdminPassword 

c6373f386b234b6abd45fdc68c86693c

登录后取消推荐插件安装

然后进入插件中心https://jenkins.chenjie.info/pluginManager/

https://jenkins.chenjie.info/manage/pluginManager/advanced 升级站点中更新插件镜像源

https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json

安装插件

Localization: Chinese

Untitled

Pipeline

Untitled

Kubernetes

Untitled

配置kubernetes集群

访问配置界面https://jenkins.chenjie.info/configureClouds/

Untitled

kubernetes cloud details配置项

Untitled

Untitled

pod template 配置项

Untitled

jenkins/jnlp-slave:4.13.3-1-jdk11 ,镜像jdk 版本需要和master的保持一致,否则会有报错

Untitled

Untitled

因为集群用的是containerd,而后续推送镜像需要docker danmen , 这里创建一个docker-dind服务,

其中需要按需修改镜像仓库地址

docker-dind.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    app: docker-dind
  name: docker-dind-data
  namespace: kube-ops
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: docker-dind
  namespace: kube-ops
  labels:
    app: docker-dind
spec:
  selector:
    matchLabels:
      app: docker-dind
  template:
    metadata:
      labels:
        app: docker-dind
    spec:
      containers:
        - image: docker:dind
          name: docker-dind
          args:
          - --insecure-registry=harbor.chenjie.info #指定Harbor仓库跳过ssl认证
          - --registry-mirror=https://ot2k4d59.mirror.aliyuncs.com/  #指定一个镜像加速器地址
          env:
            - name: DOCKER_DRIVER
              value: overlay2
            - name: DOCKER_HOST
              value: tcp://0.0.0.0:2375
            - name: DOCKER_TLS_CERTDIR   # 禁用 TLS(最好别禁用)
              value: ""
          volumeMounts:
            - name: docker-dind-data-vol # 持久化docker根目录
              mountPath: /var/lib/docker/
          ports:
            - name: daemon-port
              containerPort: 2375
          securityContext:
            privileged: true # 需要设置成特权模式
      volumes:
        - name: docker-dind-data-vol
          persistentVolumeClaim:
            claimName: docker-dind-data
---
apiVersion: v1
kind: Service
metadata:
  name: docker-dind
  namespace: kube-ops
  labels:
    app: docker-dind
spec:
  ports:
    - port: 2375
      targetPort: 2375
  selector:
    app: docker-dind

host path volume:

Untitled

Untitled

执行任务

新建任务,选择第一个

Untitled

填入上面配置的pod 模版中的标签,表示使用上面的pod 模版创建slave pod

Untitled

Untitled

Untitled

创建后点击左侧菜单 立即构建

在控制台输出查看

Started by user admin
Running as SYSTEM
Agent jenkins-agent-q2sbt is provisioned from template null
---
apiVersion: "v1"
kind: "Pod"
metadata:
  labels:
    jenkins: "slave"
    jenkins/label-digest: "47251a1e88aa724b346b9dbc4020dfdc5c577e17"
    jenkins/label: "chenjie-jnlp"
  name: "jenkins-agent-q2sbt"
  namespace: "kube-ops"
spec:
  containers:
  - env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "jenkins.kube-ops.svc.cluster.local:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "jenkins-agent-q2sbt"
    - name: "DOCKER_HOST"
      value: "tcp://docker-dind:2375"
    - name: "JENKINS_NAME"
      value: "jenkins-agent-q2sbt"
    - name: "JENKINS_AGENT_WORKDIR"
      value: "/home/jenkins/agent"
    - name: "JENKINS_URL"
      value: "http://jenkins.kube-ops.svc.cluster.local:8080/"
    image: "jenkins/jnlp-slave:4.13.3-1-jdk11"
    imagePullPolicy: "IfNotPresent"
    name: "jnlp"
    resources: {}
    tty: false
    volumeMounts:
    - mountPath: "/root/.kube"
      name: "volume-0"
      readOnly: false
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
      readOnly: false
    workingDir: "/home/jenkins/agent"
  hostNetwork: false
  nodeSelector:
    kubernetes.io/os: "linux"
  restartPolicy: "Never"
  serviceAccountName: "jenkins"
  volumes:
  - hostPath:
      path: "/root/.kube"
    name: "volume-0"
  - emptyDir:
      medium: ""
    name: "workspace-volume"

Building remotely on jenkins-agent-q2sbt (chenjie-jnlp) in workspace /home/jenkins/agent/workspace/fdfdfa
[chenjie01] $ /bin/sh -xe /tmp/jenkins4625908416244423944.sh
+ java -version
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK-11.0.11+9 (build 11.0.11+9, mixed mode)
Finished: SUCCESS

参考

  1. https://github.com/rancher/local-path-provisioner
  2. https://www.jenkins-zh.cn/tutorial/management/plugin/update-center/
  3. https://developer.aliyun.com/article/1005052#slide-11
  4. https://docs.youdianzhishi.com/k8s/devops/jenkins/

发表回复

您的电子邮箱地址不会被公开。 必填项已用 * 标注

此站点使用Akismet来减少垃圾评论。了解我们如何处理您的评论数据