kubernetes实现jenkins slave动态伸缩

  • 在Kubernetes中使用Jenkins的优点:
  1. Master服务高可用:当Jenkins Master出现故障时,Kubernetes 会自动创建一个新的Jenkins Master容器,并且将Volume分配给新创建的容器,保证数据不丢失,从而达到集群服务高可用。
  2. Slave动态伸缩:合理使用资源,每次运行Job时,会自动创建一个Jenkins Slave,Job完成后,Slave自动注销并删除容器,资源自动释放,而且Kubernetes会根据每个资源使用情况,动态分配Slave到空闲的节点上创建,降低出现因某节点资源利用率高,还排队等待在该节点的情况。
  3. 扩展性好:当Kubernetes集群的资源严重不足而导致Job排队等待时,可以很容易的添加一个Kubernetes Node到集群中,从而实现扩展。
  • Kubernetes搭建jenkins集群

当Jenkins Master 接收到Build请求时,会根据配置的Label动态创建一个运行在Pod中的Jenkins Slave并注册到Master上,当运行完Job后,这个Slave会被注销并且这个Pod也会自动删除,恢复到最初状态

部署 jenkins

部署思路

  1. 在jenkins-master里使用Kubernetes plugin创建进行slave的动态伸缩
  2. 使用nfs存储做为挂载jenkins-master的jenkins_home目录,构建时slave的maven缓存的m2目录,保留slave每次构建产生的数据(workspace目录中的每个job)

所需镜像

jenkins:2.276
jenkins/inbound-agent:4.3-4
tomcat:8.5.64-jdk8
docker:20.10.6
maven:3.6-jdk-8
bitnami/kubectl:1.18.18

部署

部署方式可以使用Kubernetes plugin官网 提供的部署yaml,咱们使用自定义的deployment

service-account.yml 此文件用于创建k8s的rbac,授权给后面的Jenkins应用可以创建和删除slave的pod

apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
namespace: demo
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: jenkins
rules:
- apiGroups: ["extensions", "apps"]
resources: ["deployments"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["services"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: jenkins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins
namespace: demo
kind: Deployment
apiVersion: apps/v1
metadata:
name: jenkins
namespace: demo
labels:
name: jenkins
spec:
replicas: 1
selector:
matchLabels:
name: jenkins
template:
metadata:
labels:
app: jenkins
name: jenkins
spec:
serviceAccount: jenkins
volumes:
- name: data
persistentVolumeClaim:
claimName: jenkins-home
containers:
- name: jenkins
image: 10.166.33.110/infra/jenkins:v2.276 # 官方镜像为jenkins为jenkins:v2.276,为了节省下载时间已经push到自己到harbor仓库
imagePullPolicy: Always
ports:
- containerPort: 8080
name: web
protocol: TCP
- containerPort: 50000 # agent连接端口
name: agent
protocol: TCP
resources:
limits:
cpu: 2048m
memory: 4096Mi
requests:
cpu: 512m
memory: 1024Mi
livenessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 5
readinessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 5
env:
- name: JAVA_OPTS
value: "-Xms2G -Xmx2G -Duser.timezone=Asia/Shanghai"
- name: TRY_UPGRADE_IF_NO_MARKER
value: "true"
volumeMounts:
- name: data
mountPath: /var/jenkins_home
securityContext: # root权限
runAsUser: 0
imagePullSecrets:
- name: harbor
kind: Service
apiVersion: v1
metadata:
name: jenkins
namespace: demo
spec:
ports:
- name: web
port: 8080
targetPort: 8080
protocol: TCP
- name: agent
port: 50000
targetPort: 50000
protocol: TCP
selector:
app: jenkins
type: NodePort
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-home
namespace: demo
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
storageClassName: nfs
volumeMode: Filesystem
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-agent
namespace: demo
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
storageClassName: nfs
volumeMode: Filesystem
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-m2
namespace: demo
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
storageClassName: nfs
volumeMode: Filesystem

查看创建状态

# kubectl get po -ndemo -l app=jenkins
NAME READY STATUS RESTARTS AGE
jenkins-7d96c568cf-trvfs 1/1 Running 0 24h
# kubectl get pvc -ndemo |grep jenkins
jenkins-agent Bound pvc-1168ca97-4765-492b-a1ac-6c98d4c2df60 1Gi RWX nfs 25m
jenkins-home Bound pvc-d7fdfcde-da3c-44dd-b882-5f49b7ee5a9b 1Gi RWX nfs 2d5h
jenkins-m2 Bound pvc-3c9d2f99-e239-45f1-942a-6874058c2b68 1Gi RWX nfs 29h

查看jenkins密码

# kubectl exec -it jenkins-7d96c568cf-trvfs -n demo -- cat /var/jenkins_home/secrets/initialAdminPassword

jenkins 配置

插件配置

更换插件源

系统管理=> 插件管理=> 高级 => https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json #清华源
安装插件

  • 选择系统管理 => 插件管理 => 可选插件 => Blue Ocean
  • 选择系统管理 => 插件管理 => 可选插件 => Kubernetes

Cloud配置

  • 后端master节点生成证书

打开~/.kube/config文件

复制certificate-authority-data的内容,运行以下命令生成client.crt
# echo "<certificate-authority-data>" | base64 -d > ca.crt
复制client-certificate-data的内容,运行以下命令生成client.crt
# echo "<client-certificate-data>" | base64 -d > client.crt
复制client-key-data的内容,运行以下命令生成client.key
# echo "<client-key-data>" | base64 -d > client.key

再根据前面步骤生成的ca.crt, client.crt和client.key来生成PKCS12格式的cert.pfx
以下命令运行时,需要输入4位以上的密码
# openssl pkcs12 -export -out cert.pfx -inkey client.key -in client.crt -certfile ca.crt
Enter Export Password:
Verifying - Enter Export Password:
  • jenkins创建证书凭证

选择 Certificate 类型 添加证书

  • 配置Kubernetes集群

系统配置拉到最后会看见一个Cloud

配置jenkins地址

gitlab配置公钥

代码仓库:代码仓库:https://github.com/wq-h/demo-2048.git

  • 创建密钥
# ssh-keygen -t rsa -b 2048 -C "weiqun_h@163.com" -N "" -f /root/.ssh/id_rsa
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:lo1dX2Kj+5Z8n7LiICLc9yYJdkhlY9W2YpNsTn/pmrI weiqun_h@163.com
The key's randomart image is:
+---[RSA 2048]----+
| ... |
| = o |
| + o o o + .|
| . & o + + |
| . . S * . o |
| . .+ o . . + |
| o.ooo.. +. . |
| . oooo...++ o|
| oE++oo+oo|
+----[SHA256]-----+
  • gitlab配置公钥

  • jenkins 创建凭据

选择ssh username with private key类型 添加私钥

jenkins动态slave测试

pipeline文件

def label = "jenkins-slave-${UUID.randomUUID().toString()}" //
podTemplate(label: label, cloud: 'kubernetes') {
node(label) {
stage('Run shell') {
sh 'sleep 10s'
sh 'echo hello world.'
}
}
}

配置java流水线

前期准备

  • .docker/config.json
# docker login -u <用户名> -p <密码> 10.166.33.110
# kubectl create secret generic jenkins-docker-cfg -n demo --from-file=/root/.docker/config.json
secret/jenkins-docker-cfg created
  • .kube/config
# kubectl create secret generic jenkins-k8s-cfg -n demo --from-file=/root/.kube/config
secret/jenkins-k8s-cfg created

jenkins配置模板

Kubernetes Plugin

  • 配置pod模板

系统配置 => Cloud => Pod Templates

  • 配置容器模板

jnlp agent镜像

maven镜像

docker镜像

kubectl镜像

  • 配置存储卷

本地文件挂载

pvc挂载

secret挂载

创建jenkins项目

  • 创建pipeline项目

  • 参数化构建工程,选择字符参数

名称: APP_NAME
默认值: demo-2048
描述: 项目名称,用于创建deployment名称,例如demo-2048

名称: APP_NS
默认值: demo
描述: namespace名称,deployment资源创建指定的ns

名称: GIT_VER
默认值: master
描述: 项目所在git中央仓库对应项目的分支或者版本号,例如master分支:master,commit ID:b8028aae

名称: GIT_REPO
默认值: ssh://git@10.166.33.116:19922/root/demo-2048.git
描述: 项目所在的git中央仓库的地址

名称: MVN_DIR
默认值: ./
描述: 编译项目目录,默认为项目的根目录

名称: MVN_CMD
默认值: mvn clean package -Dmaven.test.skip=true
描述: 执行mvn编译所用的命令

名称: IMAGE_NAME
默认值: demo/demo-2048
描述: docker镜像名称,格式:<仓库名>/<镜像名> 例如:demo/demo-2048

名称: IMAGE_REPO
默认值: 10.166.33.110
描述: docker镜像仓库名称

pipeline文件

// 流水线的最外层结构,代表整条pipeline,包含着pipeline完整逻辑
pipeline {
// 环境变量的定义
environment {
IMAGE="${params.IMAGE_REPO}/${params.IMAGE_NAME}:${params.GIT_VER}-${BUILD_NUMBER}" // 通过 ${params.xxx} 的方式对此参数进行引用
APP_NS="${params.APP_NS}"
APP_NAME="${params.APP_NAME}"
}
// pipeline中单独指令,用于指定流水的执行位置,它可能是代表着slave主机的某个物理机、虚拟机或者容器
agent {
node {
// label设置为cloud配置的pod 模板所定义的标签
label 'jenkins-mvn'
}
}
// 用于包含所有的stage的定义
stages {
// 阶段,代表流水线的一个单独的功能完成时期,例如编译等
stage('检出代码') {
// 步骤,用于在stage中定义完成该阶段功能所需经历的一系列步骤
steps {
// 递归删除WORKSPACE下的文件和文件夹,避免缓存导致构建问题
deleteDir()
// credentialsId为ssh 凭据的id
checkout([$class: 'GitSCM', branches: [[name: "${params.GIT_VER}"]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: "gitlabauth", url: "${params.GIT_REPO}"]]])
}
}
stage('构建代码') {
steps{
// 用于创建容器的容器模板
container('maven') {
sh "cd ${params.MVN_DIR} && ${params.MVN_CMD}"
sh "cd target && jar -xf ${params.APP_NAME}.war"
}
}
// post定义将在pipeline运行或stage结束时运行的操作
post {
// 成功之后提取制品
success {
archiveArtifacts "target/*.war"
}
}
}
stage('构建镜像') {
steps {
container('docker') {
sh "cp ./Dockerfiles/Dockerfile ./target"
sh "cd ./target && docker build -t ${params.IMAGE_REPO}/${params.IMAGE_NAME}:${params.GIT_VER}-${BUILD_NUMBER} ."
}
}
}
stage('推送镜像') {
steps {
container('docker') {
sh "docker push ${params.IMAGE_REPO}/${params.IMAGE_NAME}:${params.GIT_VER}-${BUILD_NUMBER}"
}
}
}
stage('更新服务') {
steps {
container('kubectl') {
# envsubst 将环境变量传递给文件
sh "envsubst < template/${params.APP_NAME}.yaml |kubectl --kubeconfig=/home/jenkins/agent/.kube/config apply -f -"
}
}
}
}
}

执行结果

# kubectl get po -ndemo|egrep 'demo-2048|jenkins-mvn'
demo-2048-6b7ffc87bc-9jd7m 1/1 Running 0 6m14s
jenkins-mvn-qq4n9 4/4 Running 0 6m43s