前置说明
本文档基于kubernetes 1.22版本使用，其他版本可能存在版本兼容问题。
前置准备
rancher local-path部署
为Jenkins提供动态pv
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.24/deploy/local-path-storage.yaml部署后
kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-path rancher.io/local-path Delete WaitForFirstConsumer false 22dTraefik安装和域名配置
为Jenkins提供域名入口支持
Jenkins部署
部署资源清单jenkins.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-pvc
namespace: kube-ops
spec:
storageClassName: local-path # 指定一个可用的 storageclass
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
namespace: kube-ops
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: jenkins
rules:
- apiGroups: ["extensions", "apps"]
resources: ["deployments", "ingresses"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["services"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
- apiGroups: [""]
resources: ["pods/log", "events"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: jenkins
namespace: kube-ops
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins
namespace: kube-ops
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: jenkins
namespace: kube-ops
spec:
selector:
matchLabels:
app: jenkins
template:
metadata:
labels:
app: jenkins
spec:
serviceAccount: jenkins
initContainers:
- name: fix-permissions
image: busybox:1.35.0
command: ["sh", "-c", "chown -R 1000:1000 /var/jenkins_home"]
securityContext:
privileged: true
volumeMounts:
- name: jenkinshome
mountPath: /var/jenkins_home
containers:
- name: jenkins
image: jenkins/jenkins:2.402
imagePullPolicy: IfNotPresent
env:
- name: JAVA_OPTS
value: -Dhudson.model.DownloadService.noSignatureCheck=true
- name: GIT_SSL_NO_VERIFY #拉取代码时跳过ssl认证
value: "true"
ports:
- containerPort: 8080
name: web
protocol: TCP
- containerPort: 50000
name: agent
protocol: TCP
resources:
limits:
cpu: 1500m
memory: 2048Mi
requests:
cpu: 1500m
memory: 2048Mi
readinessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12
volumeMounts:
- name: jenkinshome
mountPath: /var/jenkins_home
volumes:
- name: jenkinshome
persistentVolumeClaim:
claimName: jenkins-pvc
---
apiVersion: v1
kind: Service
metadata:
name: jenkins
namespace: kube-ops
labels:
app: jenkins
spec:
selector:
app: jenkins
ports:
- name: web
port: 8080
targetPort: web
- name: agent
port: 50000
targetPort: agent
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: jenkins
namespace: kube-ops
spec:
entryPoints:
- websecure
routes:
- match: Host(jenkins.chenjie.info) # 指定域名
kind: Rule
services:
- name: jenkins
port: 8080
tls:
secretName: jenkins-ssl上面清单中使用了https协议作为入口访问，需要用到tls，这里使用自签名证书为例
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=jenkins.chenjie.info"
kubectl create ns kube-ops
kubectl create secret tls jenkins-ssl -n kube-ops --cert=tls.crt --key=tls.key执行Jenkins部署指令
kubectl apply -f jenkins.yaml访问Jenkins web 服务
部署后在本地hosts 映射域名地址，访问https://jenkins.chenjie.info
密码获取方式如下
kubectl get pods -n kube-ops
NAME READY STATUS RESTARTS AGE
jenkins-7c69677d77-49rp2 1/1 Running 3 (15h ago) 15h
kubectl exec -it jenkins-7c69677d77-49rp2 -n kube-ops -- cat /var/jenkins_home/secrets/initialAdminPassword
c6373f386b234b6abd45fdc68c86693c登录后取消推荐插件安装
然后进入插件中心https://jenkins.chenjie.info/pluginManager/
在https://jenkins.chenjie.info/manage/pluginManager/advanced 升级站点中更新插件镜像源
https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json
安装插件
Localization: Chinese
Pipeline
Kubernetes
配置kubernetes集群
访问配置界面https://jenkins.chenjie.info/configureClouds/
kubernetes cloud details配置项
pod template 配置项
jenkins/jnlp-slave:4.13.3-1-jdk11 ，镜像jdk 版本需要和master的保持一致，否则会有报错
因为集群用的是containerd,而后续推送镜像需要docker danmen , 这里创建一个docker-dind服务，
其中需要按需修改镜像仓库地址
docker-dind.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
app: docker-dind
name: docker-dind-data
namespace: kube-ops
spec:
accessModes:
- ReadWriteOnce
storageClassName: local-path
resources:
requests:
storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: docker-dind
namespace: kube-ops
labels:
app: docker-dind
spec:
selector:
matchLabels:
app: docker-dind
template:
metadata:
labels:
app: docker-dind
spec:
containers:
- image: docker:dind
name: docker-dind
args:
- --insecure-registry=harbor.chenjie.info #指定Harbor仓库跳过ssl认证
- --registry-mirror=https://ot2k4d59.mirror.aliyuncs.com/ #指定一个镜像加速器地址
env:
- name: DOCKER_DRIVER
value: overlay2
- name: DOCKER_HOST
value: tcp://0.0.0.0:2375
- name: DOCKER_TLS_CERTDIR # 禁用 TLS（最好别禁用）
value: ""
volumeMounts:
- name: docker-dind-data-vol # 持久化docker根目录
mountPath: /var/lib/docker/
ports:
- name: daemon-port
containerPort: 2375
securityContext:
privileged: true # 需要设置成特权模式
volumes:
- name: docker-dind-data-vol
persistentVolumeClaim:
claimName: docker-dind-data
---
apiVersion: v1
kind: Service
metadata:
name: docker-dind
namespace: kube-ops
labels:
app: docker-dind
spec:
ports:
- port: 2375
targetPort: 2375
selector:
app: docker-dindhost path volume:
执行任务
新建任务，选择第一个
填入上面配置的pod 模版中的标签，表示使用上面的pod 模版创建slave pod
创建后点击左侧菜单 立即构建
在控制台输出查看
Started by user admin
Running as SYSTEM
Agent jenkins-agent-q2sbt is provisioned from template null
---
apiVersion: "v1"
kind: "Pod"
metadata:
labels:
jenkins: "slave"
jenkins/label-digest: "47251a1e88aa724b346b9dbc4020dfdc5c577e17"
jenkins/label: "chenjie-jnlp"
name: "jenkins-agent-q2sbt"
namespace: "kube-ops"
spec:
containers:
- env:
- name: "JENKINS_SECRET"
value: "********"
- name: "JENKINS_TUNNEL"
value: "jenkins.kube-ops.svc.cluster.local:50000"
- name: "JENKINS_AGENT_NAME"
value: "jenkins-agent-q2sbt"
- name: "DOCKER_HOST"
value: "tcp://docker-dind:2375"
- name: "JENKINS_NAME"
value: "jenkins-agent-q2sbt"
- name: "JENKINS_AGENT_WORKDIR"
value: "/home/jenkins/agent"
- name: "JENKINS_URL"
value: "http://jenkins.kube-ops.svc.cluster.local:8080/"
image: "jenkins/jnlp-slave:4.13.3-1-jdk11"
imagePullPolicy: "IfNotPresent"
name: "jnlp"
resources: {}
tty: false
volumeMounts:
- mountPath: "/root/.kube"
name: "volume-0"
readOnly: false
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
workingDir: "/home/jenkins/agent"
hostNetwork: false
nodeSelector:
kubernetes.io/os: "linux"
restartPolicy: "Never"
serviceAccountName: "jenkins"
volumes:
- hostPath:
path: "/root/.kube"
name: "volume-0"
- emptyDir:
medium: ""
name: "workspace-volume"
Building remotely on jenkins-agent-q2sbt (chenjie-jnlp) in workspace /home/jenkins/agent/workspace/fdfdfa
[chenjie01] $ /bin/sh -xe /tmp/jenkins4625908416244423944.sh
+ java -version
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK-11.0.11+9 (build 11.0.11+9, mixed mode)
Finished: SUCCESS