Jenkins Pipeline為Kubernetes應(yīng)用部署增加狀態(tài)檢測腳本優(yōu)化
更新時間：2023年12月21日 16:52:05 作者：對你無可奈何
這篇文章主要為大家介紹了Jenkins Pipeline為Kubernetes應(yīng)用部署增加狀態(tài)檢測腳本優(yōu)化示例詳解,有需要的朋友可以借鑒參考下,希望能夠有所幫助,祝大家多多進(jìn)步,早日升職加薪
引言
在軟件部署的世界中，Jenkins已經(jīng)成為自動化流程的代名詞。不斷變化的技術(shù)環(huán)境要求我們持續(xù)改進(jìn)部署流程以滿足現(xiàn)代應(yīng)用部署的需要。在本篇博客中，作為一位資深運維工程師，我將分享如何將Jenkins Pipeline進(jìn)化至不僅能支持部署應(yīng)用直至Running狀態(tài)檢測，同時也能兼顧Deployment和StatefulSet資源的輪詢更新，并詳細(xì)介紹滾動更新策略的配置方法。
初始Jenkins Pipeline分析
參照前文：Jenkins Pipeline 腳本優(yōu)化實踐：從繁瑣到簡潔，初始化pipeline如下：
pipeline {
agent none // Use none at the top level, each stage will define its own agent.
environment {
REGISTRY = "swr.cn-north-4.myhuaweicloud.com/master-metaspace"
KUBE_CONFIG = "--namespace=master-metaspace --context=master"
KUBE_YAML_PATH = "/home/jenkins/workspace/yaml/master-metaspace"
// Assume that 'data' is defined elsewhere or injected as a parameter.
BASE_WORKSPACE = "xxxxxxx" // 定義一個基礎(chǔ)工作空間路徑
}
stages {
stage("GetCode") {
agent { label "build01" }
steps {
script {
checkout scm: [
$class: 'GitSCM',
branches: [[name: env.branchName]],
extensions: [[$class: 'CloneOption', depth: 1, noTags: false, shallow: true]],
userRemoteConfigs: [[credentialsId: 'xxxx', url: env.gitHttpURL]]
]
}
}
}
stage("Docker Builds") {
parallel {
stage('Build dataloader-game-ucenter') {
agent { label "build01" }
when { environment name: 'dataloader', value: 'true' }
steps {
buildAndPushDockerImage("dataloader-game-ucenter", env.data, env.BASE_WORKSPACE)
}
}
stage('Build datawriter-game-ucenter') {
agent { label "build01" }
when { environment name: 'datawriter', value: 'true' }
steps {
buildAndPushDockerImage("datawriter-game-ucenter", env.data, env.BASE_WORKSPACE)
}
}
stage('Build game-ucenter') {
agent { label "build01" }
when { environment name: 'game-ucenter', value: 'true' }
steps {
buildAndPushDockerImage("game-ucenter", env.data, env.BASE_WORKSPACE)
}
}
}
}
stage('Development Deployment') {
parallel {
stage("Deploy datawriter-game-ucenter") {
when { environment name: 'datawriter-game-ucenter', value: 'true' }
agent { label "huaweiyun-xx" }
steps {
deployToKubernetes("datawriter-game-ucenter")
}
}
stage("Deploy dataloader-game-ucenter") {
when { environment name: 'dataloader', value: 'true' }
agent { label "huaweiyun-xx" }
steps {
deployToKubernetes("dataloader-game-ucenter")
}
}
stage("Deploy game-ucenter") {
when { environment name: 'game-ucenter', value: 'true' }
agent { label "huaweiyun-xx" }
steps {
deployToKubernetes("game-ucenter")
}
}
}
}
}
}
// Define methods outside pipeline to avoid repetition
def buildAndPushDockerImage(String imageName, String tag, String workspacePath) {
sh "cd ${workspacePath} && echo 'Current directory: \$(pwd)'" // 使用基礎(chǔ)工作空間變量
sh "cd ${workspacePath}/${imageName}&& docker build --build-arg NODE_ENV=$imageName -t $REGISTRY/$imageName:$tag ."
withCredentials([usernamePassword(credentialsId: 'hw-registry', passwordVariable: 'dockerPassword', usernameVariable: 'dockerUser')]) {
sh "docker login -u $dockerUser -p $dockerPassword $REGISTRY"
sh "docker push $REGISTRY/$imageName:$tag"
}
}
def deployToKubernetes(String kubernetesComponent) {
String templateFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.tpl"
String outputFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.yaml"
sh "sed -e 's/{data}/$data/g' $templateFile > $outputFile"
sh "sudo kubectl apply -f $outputFile $KUBE_CONFIG"
}初始的Jenkins Pipeline定義了一個基本的CI/CD流程，涵蓋了代碼拉取、Docker鏡像構(gòu)建、推送及在Kubernetes環(huán)境中的部署。然而，流程中缺少了對部署狀態(tài)的檢查，這是在確保部署穩(wěn)定性方面至關(guān)重要的一個環(huán)節(jié)。
進(jìn)化 I：探針引入Deployment部署
現(xiàn)代應(yīng)用部署不僅僅需要一個“部署到Kubernetes”的指令，更需要在部署后進(jìn)行健康檢查。對于Deployment類型的應(yīng)用來說，我們需要在所有Pods運行并處于READY狀態(tài)后才認(rèn)為部署成功。
狀態(tài)檢測方法介紹
為此，我們引入了checkKubernetesResourceStatus方法來檢查資源的狀態(tài)。該方法通過kubectl的get命令和jsonpath查詢輸出來輪詢檢查ready副本數(shù)。如果指定時間內(nèi)資源不達(dá)狀態(tài)，則流程失敗。
Jenkinsfile變更詳解：
引入checkKubernetesResourceStatus方法來檢測deployment各個階段部署的狀態(tài)。
def checkKubernetesResourceStatus(String deploymentName, String namespace) {
int attempts = 30 // Set the number of retry attempts
int sleepTime = 10 // Set the sleep time between attempts in seconds
String readyReplicasJsonPath = ".status.readyReplicas"
for (int i = 1; i <= attempts; i++) {
// Check the deployment status
String statusCheck = sh (
script: "kubectl get deployment ${deploymentName} --namespace=${namespace} -o jsonpath=\"{${readyReplicasJsonPath}}\"",
returnStdout: true
).trim()
// If the number of ready replicas is not empty and greater than 0
if (statusCheck && statusCheck.isInteger() && statusCheck.toInteger() > 0) {
echo "Deployment ${deploymentName} is ready."
return
} else {
echo "Waiting for Deployment ${deploymentName} to be ready. Attempt ${i}/${attempts}"
sleep sleepTime
}
}
error "Deployment ${deploymentName} did not become ready after ${attempts} attempts"
}以Deploy game-ucenter stage為例：
stage("Deploy game-ucenter") {
when { environment name: 'game-ucenter', value: 'true' }
agent { label "xxxx" }
steps {
deployToKubernetes("game-ucenter")
checkKubernetesResourceStatus("game-ucenter", "master-metaspace")
}
}game-ucenter模板文件如下：
apiVersion: apps/v1
kind: Deployment
metadata:
name: game-ucenter
spec:
replicas: 1
selector:
matchLabels:
app: game-ucenter
template:
metadata:
labels:
app: game-ucenter
spec:
containers:
- name: game-ucenter
image: xxxxxx/xxxx/game-ucenter:{data}
envFrom:
- configMapRef:
name: deploy
ports:
- containerPort: 80
resources:
requests:
memory: "4096M"
cpu: "2000m"
limits:
memory: "4096M"
cpu: "2000m"
livenessProbe:
httpGet:
scheme: HTTP
path: /test.html
port: 80
initialDelaySeconds: 20
periodSeconds: 120
successThreshold: 1
failureThreshold: 3
readinessProbe:
httpGet:
scheme: HTTP
path: /test.html
port: 80
initialDelaySeconds: 20
periodSeconds: 120
imagePullSecrets:
- name: xxxx
---
apiVersion: v1
kind: Service
metadata:
name: game-ucenter
labels:
app: game-ucenter
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: game-ucenter
嘗試修改以下replicas: 3 重新運行以下pipeline：
也沒有問題，pipeline 腳本有效！
進(jìn)化 II：兼容StatefulSet的健康檢查
考慮到某些應(yīng)用可能采用StatefulSet作為工作負(fù)載類型，我們必須確保Jenkins Pipeline能夠針對不同的工作負(fù)載類型執(zhí)行健康檢查。
狀態(tài)檢測兼容性改進(jìn)
為了適配StatefulSet，我們對checkKubernetesResourceStatus方法做了略微修改，使其可以接受一個resourceType參數(shù)來區(qū)分資源類型，進(jìn)而查詢對應(yīng)的狀態(tài)字段，代碼片段如下：
def checkKubernetesResourceStatus(String resourceName, String namespace, String resourceType) {
int attempts = 30 // Set the number of retry attempts
int sleepTime = 10 // Set the sleep time between attempts in seconds
String readyReplicasJsonPath = resourceType == "deployment" ? ".status.readyReplicas" : ".status.readyReplicas"
for (int i = 1; i <= attempts; i++) {
// Check the resource status
String statusCheck = sh (
script: "kubectl get ${resourceType} ${resourceName} --namespace=${namespace} -o jsonpath=\"{${readyReplicasJsonPath}}\"",
returnStdout: true
).trim()
// If the number of ready replicas is not empty and equal to the desired number
if (statusCheck && statusCheck.isInteger() && statusCheck.toInteger() > 0) {
echo "${resourceType} ${resourceName} is ready."
return
} else {
echo "Waiting for ${resourceType} ${resourceName} to be ready. Attempt ${i}/${attempts}"
sleep(sleepTime)
}
}
error "${resourceType} ${resourceName} did not become ready after ${attempts} attempts"
}修改game-ucenter stage:
stage("Deploy game-ucenter") {
when { environment name: 'game-ucenter', value: 'true' }
agent { label "k8s-node-06" }
steps {
deployToKubernetes("game-ucenter")
checkKubernetesResourceStatus("game-ucenter", "master-metaspace", "deployment")
}
}創(chuàng)建一個statefulset datawriter-game-ucenter stage:
stage("Deploy datawriter-game-ucenter") {
when { environment name: 'datawriter-game-ucenter', value: 'true' }
agent { label "xxxxx" }
steps {
deployToKubernetes("datawriter-game-ucenter")
checkKubernetesResourceStatus("datawriter-game-ucenter", "master-metaspace", "statefulset")
}
}
注意:我這里截圖還是用了game-ucenter做的測試，其實我想用我的datawriter-game-ucenter，but這個服務(wù)是一個node應(yīng)用沒有沒有livenessProbe readinessProbe，所以截圖我還是使用了game-ucenter!
進(jìn)化 III：引入滾動更新策略配置和檢測
當(dāng)我們更新Deployment資源時，通常會采用滾動更新策略，以逐步替換舊Pods，最小化部署時的中斷。
更新策略檢測邏輯
def checkDeploymentUpdateStatus(String deploymentName, String namespace) {
int attempts = 30 // Set the number of retry attempts
int sleepTime = 10 // Set the sleep time between attempts in seconds
echo "Checking the update status of Deployment: ${deploymentName}"
for (int i = 1; i <= attempts; i++) {
String updateStatus = sh (
script: "kubectl rollout status deployment/${deploymentName} --namespace=${namespace}",
returnStdout: true
).trim()
if (updateStatus.contains("successfully rolled out")) {
echo "Update status: ${updateStatus}"
return
} else {
echo "Waiting for Deployment ${deploymentName} to successfully roll out. Attempt ${i}/${attempts}"
sleep(sleepTime)
}
}
error "Deployment ${deploymentName} did not successfully roll out after ${attempts} attempts"
}- 我們增加了
checkDeploymentUpdateStatus方法，該方法通過kubectl命令rollout status監(jiān)控Deployment的更新狀態(tài)。 - 當(dāng)檢測到
successfully rolled out時，表示滾動更新成功。 - 如果在給定時間內(nèi)更新沒有成功，則流程將失敗。
繼續(xù)考慮一下如果statefulset多實例呢？不想寫兩個了整合成一個方法如下：
def checkRolloutStatus(String resourceName, String namespace, String resourceType) {
int attempts = 30 // Set the number of retry attempts
int sleepTime = 10 // Set the sleep time between attempts in seconds
if (!(resourceType in ["deployment", "statefulset"])) {
error "Unknown resource type: ${resourceType}. Only 'deployment' and 'statefulset' are supported."
}
echo "Checking the update status of ${resourceType} '${resourceName}' in namespace '${namespace}'"
for (int i = 1; i <= attempts; i++) {
String rolloutCommand = "kubectl rollout status ${resourceType}/${resourceName} --namespace=${namespace}"
String updateStatus = sh (
script: rolloutCommand,
returnStdout: true
).trim()
if (updateStatus.contains("successfully rolled out") || updateStatus.contains("partitioned roll out complete")) {
echo "Update status: ${updateStatus}"
return
} else {
echo "Waiting for ${resourceType} '${resourceName}' to successfully roll out. Attempt ${i}/${attempts}."
sleep(sleepTime)
}
}
error "${resourceType} '${resourceName}' did not successfully roll out after ${attempts} attempts in namespace '${namespace}'"
}Jenkinsfile更新實現(xiàn)
經(jīng)過上述進(jìn)化，Jenkinsfile中現(xiàn)在包含了完整的部署狀態(tài)檢查邏輯，以應(yīng)對不同類型資源的部署監(jiān)控需求。
pipeline {
agent none // Use none at the top level, each stage will define its own agent.
environment {
REGISTRY = "ccr.ccs.tencentyun.com/xxxxx"
KUBE_CONFIG = "--namespace=master-metaspace"
KUBE_YAML_PATH = "/home/jenkins/workspace/yaml/master-metaspace"
// Assume that 'data' is defined elsewhere or injected as a parameter.
BASE_WORKSPACE = "xxxxxx" // 定義一個基礎(chǔ)工作空間路徑
}
stages {
stage("GetCode") {
agent { label "build01" }
steps {
script {
checkout scm: [
$class: 'GitSCM',
branches: [[name: env.branchName]],
extensions: [[$class: 'CloneOption', depth: 1, noTags: false, shallow: true]],
userRemoteConfigs: [[credentialsId: 'xxxxx', url: env.gitHttpURL]]
]
}
}
}
stage("Docker Builds") {
parallel {
stage('Build dataloader-game-ucenter') {
agent { label "build01" }
when { environment name: 'dataloader-game-ucenter', value: 'true' }
steps {
buildAndPushDockerImage("dataloader-game-ucenter", env.data, env.BASE_WORKSPACE)
}
}
stage('Build datawriter-game-ucenter') {
agent { label "build01" }
when { environment name: 'datawriter-game-ucenter', value: 'true' }
steps {
buildAndPushDockerImage("datawriter-game-ucenter", env.data, env.BASE_WORKSPACE)
}
}
stage('Build game-ucenter') {
agent { label "build01" }
when { environment name: 'game-ucenter', value: 'true' }
steps {
buildAndPushDockerImage("game-ucenter", env.data, env.BASE_WORKSPACE)
}
}
}
}
stage('Development Deployment') {
parallel {
stage("Deploy datawriter-game-ucenter") {
when { environment name: 'datawriter-game-ucenter', value: 'true' }
agent { label "xxxx" }
steps {
deployToKubernetes("datawriter-game-ucenter")
checkKubernetesResourceStatus("datawriter-game-ucenter", "master-metaspace", "statefulset")
checkRolloutStatus("datawriter-game-ucenter", "master-metaspace", "statefulset")
}
}
stage("Deploy dataloader-game-ucenter") {
when { environment name: 'dataloader-game-ucenter', value: 'true' }
agent { label "xxxx" }
steps {
deployToKubernetes("dataloader-game-ucenter")
checkKubernetesResourceStatus("dataloader-game-ucenter", "master-metaspace", "statefulset")
}
}
stage("Deploy game-ucenter") {
when { environment name: 'game-ucenter', value: 'true' }
agent { label "xxxx" }
steps {
deployToKubernetes("game-ucenter")
checkRolloutStatus("game-ucenter", "master-metaspace", "deployment")
checkKubernetesResourceStatus("game-ucenter", "master-metaspace", "deployment")
}
}
}
}
}
}
// Define methods outside pipeline to avoid repetition
def buildAndPushDockerImage(String imageName, String tag, String workspacePath) {
sh "cd ${workspacePath} && echo 'Current directory: \$(pwd)'" // 使用基礎(chǔ)工作空間變量
sh "cd ${workspacePath}/${imageName}&& docker build --build-arg NODE_ENV=game-ucenter -t $REGISTRY/$imageName:$tag ."
withCredentials([usernamePassword(credentialsId: 'xxxxx', passwordVariable: 'dockerPassword', usernameVariable: 'dockerUser')]) {
sh "docker login -u $dockerUser -p $dockerPassword $REGISTRY"
sh "docker push $REGISTRY/$imageName:$tag"
}
}
def deployToKubernetes(String kubernetesComponent) {
String templateFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.tpl"
String outputFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.yaml"
sh "sed -e 's/{data}/$data/g' $templateFile > $outputFile"
sh "sudo kubectl apply -f $outputFile $KUBE_CONFIG"
}
def checkRolloutStatus(String resourceName, String namespace, String resourceType) {
int attempts = 30 // 設(shè)置重試次數(shù)
int sleepTime = 10 // 設(shè)置重試間隔時間（秒）
if (!(resourceType in ["deployment", "statefulset"])) {
error "未知資源類型：${resourceType}。只支持 'deployment' 和 'statefulset' 。"
}
echo "正在檢查${resourceType} '${resourceName}' 在命名空間 '${namespace}' 的更新狀態(tài)"
for (int i = 1; i <= attempts; i++) {
String rolloutCommand = "kubectl rollout status ${resourceType}/${resourceName} --namespace=${namespace}"
try {
String updateStatus = sh (
script: rolloutCommand,
returnStdout: true
).trim()
// 添加對 "partitioned roll out complete" 狀態(tài)的檢查
if (updateStatus.contains("successfully rolled out") || updateStatus.contains("partitioned roll out complete")) {
echo "更新狀態(tài)：${updateStatus}"
return
} else {
echo "等待 ${resourceType} '${resourceName}' 成功發(fā)布。嘗試次數(shù)：${i}/${attempts}。"
sleep(sleepTime)
}
} catch (Exception e) {
echo "獲取更新狀態(tài)時發(fā)生錯誤：${e.getMessage()}。嘗試次數(shù)：${i}/${attempts}。"
sleep(sleepTime)
}
}
error "${resourceType} '${resourceName}' 在命名空間 '${namespace}' 內(nèi)未能在 ${attempts} 次嘗試之后成功發(fā)布"
}
def checkKubernetesResourceStatus(String resourceName, String namespace, String resourceType) {
int attempts = 30 // Set the number of retry attempts
int sleepTime = 10 // Set the sleep time between attempts in seconds
String readyReplicasJsonPath = resourceType == "deployment" ? ".status.readyReplicas" : ".status.readyReplicas"
for (int i = 1; i <= attempts; i++) {
// Check the resource status
String statusCheck = sh (
script: "kubectl get ${resourceType} ${resourceName} --namespace=${namespace} -o jsonpath=\"{${readyReplicasJsonPath}}\"",
returnStdout: true
).trim()
// If the number of ready replicas is not empty and equal to the desired number
if (statusCheck && statusCheck.isInteger() && statusCheck.toInteger() > 0) {
echo "${resourceType} ${resourceName} is ready."
return
} else {
echo "Waiting for ${resourceType} ${resourceName} to be ready. Attempt ${i}/${attempts}"
sleep(sleepTime)
}
}
error "${resourceType} ${resourceName} did not become ready after ${attempts} attempts"
}// 更新后的Jenkins Pipeline代碼詳細(xì)定義參照本文開頭給出的代碼
總結(jié)
本篇博客通過對Jenkins Pipeline的進(jìn)化過程展開講解，展現(xiàn)了如何從簡單的部署任務(wù)轉(zhuǎn)變?yōu)橐粋€健壯且兼顧各類工作負(fù)載狀態(tài)監(jiān)測的CI/CD流程。我們強(qiáng)化了狀態(tài)檢測的邏輯，引入了更新策略的檢測，并保持了對不同Kubernetes資源類型的兼容性。這些改進(jìn)確保了自動化流程能夠與現(xiàn)代部署實踐保持同步，給運維團(tuán)隊帶來極大便利，并最大化地保障了部署的可靠性。
后記
由于篇幅限制，本篇文章未作其他更詳細(xì)演示。然而，在實際應(yīng)用中，運維團(tuán)隊可以根據(jù)自己的具體需求和環(huán)境進(jìn)一步豐富和細(xì)化每個步驟的實現(xiàn)，確保Pipeline的健壯性和高可用性，以適應(yīng)不斷變化的技術(shù)挑戰(zhàn)。以上實驗中使用了騰訊云的鏡像倉庫服務(wù)，kubernetes集群環(huán)境使用了Tke服務(wù)！
以上就是Jenkins Pipeline為Kubernetes應(yīng)用部署增加狀態(tài)檢測腳本優(yōu)化的詳細(xì)內(nèi)容，更多關(guān)于Jenkins Pipeline檢測Kubernetes的資料請關(guān)注腳本之家其它相關(guān)文章！
相關(guān)文章
基于Spring Mvc實現(xiàn)的Excel文件上傳下載示例
本篇文章主要介紹了基于Spring Mvc實現(xiàn)的Excel文件上傳下載示例，小編覺得挺不錯的，現(xiàn)在分享給大家，也給大家做個參考。一起跟隨小編過來看看吧2017-02-02
mybatis打印的sql日志不寫入到log文件的問題及解決
這篇文章主要介紹了mybatis打印的sql日志不寫入到log文件的問題及解決方案,具有很好的參考價值,希望對大家有所幫助,如有錯誤或未考慮完全的地方,望不吝賜教2023-08-08
Java中的CountDownLatch、CyclicBarrier和semaphore實現(xiàn)原理解讀
這篇文章主要介紹了Java中的CountDownLatch、CyclicBarrier和semaphore實現(xiàn)原理詳解,CountDownLatch中調(diào)用await方法線程需要等待所有調(diào)用countDown方法的線程執(zhí)行,這就很適合一個業(yè)務(wù)需要一些準(zhǔn)備條件,等準(zhǔn)備條件準(zhǔn)備好之后再繼續(xù)執(zhí)行,需要的朋友可以參考下2023-12-12
IntelliJ IDEA安裝目錄和設(shè)置目錄的說明(IntelliJ IDEA快速入門)
這篇文章主要介紹了IntelliJ IDEA安裝目錄和設(shè)置目錄的說明(IntelliJ IDEA快速入門),本文給大家介紹的非常詳細(xì)，對大家的學(xué)習(xí)或工作具有一定的參考借鑒價值，需要的朋友可以參考下2021-04-04
下面小編就為大家?guī)硪黄獪\談在JAVA項目中LOG4J的使用。小編覺得挺不錯的，現(xiàn)在就分享給大家，也給大家做個參考。一起跟隨小編過來看看吧2016-12-12
內(nèi)部類是指在一個外部類的內(nèi)部再定義一個類,下面這篇文章主要給大家介紹了關(guān)于java內(nèi)部類的最詳細(xì)詳解,文中通過圖文介紹的非常詳細(xì),需要的朋友可以參考下2022-06-06
springboot 無法掃描到父類模塊中Bean的原因及解決
這篇文章主要介紹了springboot 無法掃描到父類模塊中Bean的原因及解決方案，具有很好的參考價值，希望對大家有所幫助。如有錯誤或未考慮完全的地方，望不吝賜教2021-08-08
這篇文章主要介紹了Java中的Vector詳細(xì)解讀,Vector是實現(xiàn)了List接口的子類,其底層是一個對象數(shù)組,維護(hù)了一個elementData數(shù)組,是線程安全的,Vector類的方法帶有synchronized關(guān)鍵字,在開發(fā)中考慮線程安全中使用Vector,需要的朋友可以參考下2023-09-09
這篇文章主要介紹了@Transactional注解不起作用的原因分析及解決方案，具有很好的參考價值，希望對大家有所幫助。如有錯誤或未考慮完全的地方，望不吝賜教2021-09-09
java HttpClient傳輸json格式的參數(shù)實例講解
這篇文章主要介紹了java HttpClient傳輸json格式的參數(shù)實例講解，具有很好的參考價值，希望對大家有所幫助。一起跟隨小編過來看看吧2021-01-01