kubernetes实现jenkins slave动态伸缩
在Kubernetes中使用Jenkins的优点：
Master服务高可用：当Jenkins Master出现故障时，Kubernetes 会自动创建一个新的Jenkins Master容器，并且将Volume分配给新创建的容器，保证数据不丢失，从而达到集群服务高可用。
Slave动态伸缩：合理使用资源，每次运行Job时，会自动创建一个Jenkins Slave，Job完成后，Slave自动注销并删除容器，资源自动释放，而且Kubernetes会根据每个资源使用情况，动态分配Slave到空闲的节点上创建，降低出现因某节点资源利用率高，还排队等待在该节点的情况。
扩展性好：当Kubernetes集群的资源严重不足而导致Job排队等待时，可以很容易的添加一个Kubernetes Node到集群中，从而实现扩展。
当Jenkins Master 接收到Build请求时，会根据配置的Label动态创建一个运行在Pod中的Jenkins Slave并注册到Master上，当运行完Job后，这个Slave会被注销并且这个Pod也会自动删除，恢复到最初状态
部署 jenkins 部署思路
在jenkins-master里使用Kubernetes plugin创建进行slave的动态伸缩
使用nfs存储做为挂载jenkins-master的jenkins_home目录，构建时slave的maven缓存的m2目录，保留slave每次构建产生的数据（workspace目录中的每个job）
所需镜像 jenkins:2.276 jenkins/inbound-agent:4.3-4 tomcat:8.5.64-jdk8 docker:20.10.6 maven:3.6-jdk-8 bitnami/kubectl:1.18.18
部署 部署方式可以使用Kubernetes plugin官网 提供的部署yaml，咱们使用自定义的deployment
service-account.yml 此文件用于创建k8s的rbac，授权给后面的Jenkins应用可以创建和删除slave的pod
apiVersion: v1 kind: ServiceAccount metadata: name: jenkins namespace: demo --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: jenkins rules: - apiGroups: ["extensions" , "apps" ] resources: ["deployments" ] verbs: ["create" , "delete" , "get" , "list" , "watch" , "patch" , "update" ] - apiGroups: ["" ] resources: ["services" ] verbs: ["create" , "delete" , "get" , "list" , "watch" , "patch" , "update" ] - apiGroups: ["" ] resources: ["pods" ] verbs: ["create" ,"delete" ,"get" ,"list" ,"patch" ,"update" ,"watch" ] - apiGroups: ["" ] resources: ["pods/exec" ] verbs: ["create" ,"delete" ,"get" ,"list" ,"patch" ,"update" ,"watch" ] - apiGroups: ["" ] resources: ["pods/log" ] verbs: ["get" ,"list" ,"watch" ] - apiGroups: ["" ] resources: ["secrets" ] verbs: ["get" ] --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: jenkins roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: jenkins subjects: - kind: ServiceAccount name: jenkins namespace: demo
kind: Deployment apiVersion: apps/v1 metadata: name: jenkins namespace: demo labels: name: jenkins spec: replicas: 1 selector: matchLabels: name: jenkins template: metadata: labels: app: jenkins name: jenkins spec: serviceAccount: jenkins volumes: - name: data persistentVolumeClaim: claimName: jenkins-home containers: - name: jenkins image: 10.166 .33 .110 /infra/jenkins:v2.276 imagePullPolicy: Always ports: - containerPort: 8080 name: web protocol: TCP - containerPort: 50000 name: agent protocol: TCP resources: limits: cpu: 2048m memory: 4096Mi requests: cpu: 512m memory: 1024Mi livenessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 failureThreshold: 5 readinessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 failureThreshold: 5 env: - name: JAVA_OPTS value: "-Xms2G -Xmx2G -Duser.timezone=Asia/Shanghai" - name: TRY_UPGRADE_IF_NO_MARKER value: "true" volumeMounts: - name: data mountPath: /var/jenkins_home securityContext: runAsUser: 0 imagePullSecrets: - name: harbor
kind: Service apiVersion: v1 metadata: name: jenkins namespace: demo spec: ports: - name: web port: 8080 targetPort: 8080 protocol: TCP - name: agent port: 50000 targetPort: 50000 protocol: TCP selector: app: jenkins type: NodePort
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: jenkins-home namespace: demo spec: accessModes: - ReadWriteMany resources: requests: storage: 1Gi storageClassName: nfs volumeMode: Filesystem --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: jenkins-agent namespace: demo spec: accessModes: - ReadWriteMany resources: requests: storage: 1Gi storageClassName: nfs volumeMode: Filesystem --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: jenkins-m2 namespace: demo spec: accessModes: - ReadWriteMany resources: requests: storage: 1Gi storageClassName: nfs volumeMode: Filesystem
查看创建状态
# kubectl get po -ndemo -l app=jenkins NAME READY STATUS RESTARTS AGE jenkins-7d96c568cf-trvfs 1/1 Running 0 24h # kubectl get pvc -ndemo |grep jenkins jenkins-agent Bound pvc-1168ca97-4765-492b-a1ac-6c98d4c2df60 1Gi RWX nfs 25m jenkins-home Bound pvc-d7fdfcde-da3c-44dd-b882-5f49b7ee5a9b 1Gi RWX nfs 2d5h jenkins-m2 Bound pvc-3c9d2f99-e239-45f1-942a-6874058c2b68 1Gi RWX nfs 29h
查看jenkins密码
# kubectl exec -it jenkins-7d96c568cf-trvfs -n demo -- cat /var/jenkins_home/secrets/initialAdminPassword
jenkins 配置 插件配置 更换插件源
系统管理=> 插件管理=> 高级 => https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json #清华源
选择系统管理 => 插件管理 => 可选插件 => Blue Ocean
选择系统管理 => 插件管理 => 可选插件 => Kubernetes
Cloud配置
打开~/.kube/config文件
复制certificate-authority-data的内容，运行以下命令生成client.crt # echo "<certificate-authority-data>" | base64 -d > ca.crt 复制client-certificate-data的内容，运行以下命令生成client.crt # echo "<client-certificate-data>" | base64 -d > client.crt 复制client-key-data的内容，运行以下命令生成client.key # echo "<client-key-data>" | base64 -d > client.key 再根据前面步骤生成的ca.crt, client.crt和client.key来生成PKCS12格式的cert.pfx 以下命令运行时，需要输入4位以上的密码 # openssl pkcs12 -export -out cert.pfx -inkey client.key -in client.crt -certfile ca.crt Enter Export Password: Verifying - Enter Export Password:
选择 Certificate 类型 添加证书
系统配置拉到最后会看见一个Cloud
配置jenkins地址
gitlab配置公钥 代码仓库：代码仓库：https://github.com/wq-h/demo-2048.git
# ssh-keygen -t rsa -b 2048 -C "weiqun_h@163.com" -N "" -f /root/.ssh/id_rsa Generating public/private rsa key pair. Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:lo1dX2Kj+5Z8n7LiICLc9yYJdkhlY9W2YpNsTn/pmrI weiqun_h@163.com The key's randomart image is: +---[RSA 2048]----+ | ... | | = o | | + o o o + .| | . & o + + | | . . S * . o | | . .+ o . . + | | o.ooo.. +. . | | . oooo...++ o| | oE++oo+oo| +----[SHA256]-----+
选择ssh username with private key类型 添加私钥
jenkins动态slave测试 pipeline文件
def label = "jenkins-slave-${UUID.randomUUID().toString()}" // podTemplate(label: label, cloud: 'kubernetes' ) { node(label) { stage('Run shell') { sh 'sleep 10s' sh 'echo hello world.' } } }
配置java流水线 前期准备
# docker login -u <用户名> -p <密码> 10.166.33.110 # kubectl create secret generic jenkins-docker-cfg -n demo --from-file=/root/.docker/config.json secret/jenkins-docker-cfg created
# kubectl create secret generic jenkins-k8s-cfg -n demo --from-file=/root/.kube/config secret/jenkins-k8s-cfg created
jenkins配置模板 Kubernetes Plugin
系统配置 => Cloud => Pod Templates
jnlp agent镜像
maven镜像
docker镜像
kubectl镜像
本地文件挂载
pvc挂载
secret挂载
创建jenkins项目
名称: APP_NAME
名称: APP_NS
名称: GIT_VER
名称: GIT_REPOgit@10.166.33.116 :19922/root/demo-2048.git
名称: MVN_DIR
名称: MVN_CMD
名称: IMAGE_NAME
名称: IMAGE_REPO
pipeline文件 // 流水线的最外层结构，代表整条pipeline，包含着pipeline完整逻辑 pipeline { // 环境变量的定义 environment { IMAGE="$ {params.IMAGE_REPO }/$ {params.IMAGE_NAME }:$ {params.GIT_VER }-$ {BUILD_NUMBER }" // 通过 ${params.xxx} 的方式对此参数进行引用 APP_NS=" $ {params.APP_NS }" APP_NAME=" $ {params.APP_NAME }" } // pipeline中单独指令，用于指定流水的执行位置，它可能是代表着slave主机的某个物理机、虚拟机或者容器 agent { node { // label设置为cloud配置的pod 模板所定义的标签 label 'jenkins-mvn' } } // 用于包含所有的stage的定义 stages { // 阶段，代表流水线的一个单独的功能完成时期，例如编译等 stage('检出代码') { // 步骤，用于在stage中定义完成该阶段功能所需经历的一系列步骤 steps { // 递归删除WORKSPACE下的文件和文件夹,避免缓存导致构建问题 deleteDir() // credentialsId为ssh 凭据的id checkout([$class: 'GitSCM', branches: [[name: " $ {params.GIT_VER }"]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: " gitlabauth" , url: "${params.GIT_REPO}" ]]]) } } stage('构建代码') { steps { // 用于创建容器的容器模板 container('maven') { sh "cd ${params.MVN_DIR} && ${params.MVN_CMD}" sh "cd target && jar -xf ${params.APP_NAME}.war" } } // post定义将在pipeline运行或stage结束时运行的操作 post { // 成功之后提取制品 success { archiveArtifacts "target/*.war" } } } stage('构建镜像') { steps { container('docker') { sh "cp ./Dockerfiles/Dockerfile ./target" sh "cd ./target && docker build -t ${params.IMAGE_REPO}/${params.IMAGE_NAME}:${params.GIT_VER}-${BUILD_NUMBER} ." } } } stage('推送镜像') { steps { container('docker') { sh "docker push ${params.IMAGE_REPO}/${params.IMAGE_NAME}:${params.GIT_VER}-${BUILD_NUMBER}" } } } stage('更新服务') { steps { container('kubectl') { sh "envsubst < template/${params.APP_NAME}.yaml |kubectl --kubeconfig=/home/jenkins/agent/.kube/config apply -f -" } } } } }
执行结果 # kubectl get po -ndemo|egrep 'demo-2048|jenkins-mvn' demo-2048-6b7ffc87bc-9jd7m 1/1 Running 0 6m14s jenkins-mvn-qq4n9 4/4 Running 0 6m43s
