K8S?prometheus?operator監(jiān)控工作原理介紹
1. Prometheus Operator介紹
介紹文章:http://www.dbjr.com.cn/article/267693.htm
2016年年末,CoreOs
引入了Operator 模式,并發(fā)布了Prometheus Operator 作為Operator模式
的工作示例。Prometheus Operator
自動創(chuàng)建和管理Prometheus
監(jiān)控實例。
Prometheus Operator
的任務(wù)是使得在Kubernetes
運行Prometheus
僅可能容易,同時保留可配置性以及使Kubernetes
配置原生。
Prometheus Operator
使我們的生活更容易——部署和維護。
2. 它如何工作
為了理解這個問題,我們首先需要了解Prometheus Operator
得工作原理。
Prometheus Operator
架構(gòu)圖.
我們成功部署 Prometheus Operator
后可以看到一個新的CRDs(Custom Resource Defination):
- Prometheus,定義一個期望的
Prometheus deployment
。 - ServiceMonitor,聲明式指定應(yīng)該如何監(jiān)控服務(wù)組;
Operator
根據(jù)定義自動創(chuàng)建Prometheusscrape
配置。 - Alertmanager,定義期望的
Alertmanager deployment
。
當(dāng)服務(wù)新版本更新時,將會常見一個新Pod
。Prometheus
監(jiān)控k8s API
,因此當(dāng)它檢測到這種變化時,它將為這個新服務(wù)(pod)創(chuàng)建一組新的配置。
3. ServiceMonitor
Prometheus Operator
使用一個CRD
,叫做 ServiceMonitor 將配置抽象到目標(biāo)。
下面是個ServiceMonitor
的示例:
apiVersion: monitoring.coreos.com/v1alpha1 kind: ServiceMonitor metadata: name: frontend labels: tier: frontend spec: selector: matchLabels: tier: frontend endpoints: - port: web # 指定exporter端口,這里指定的是endpoint的名稱 interval: 10s # 刷新間隔時間
這僅僅是定義一組服務(wù)應(yīng)該如何被監(jiān)控?,F(xiàn)在我們需要定義一個包含了該ServiceMonitor
的Prometheus
實例到其配置:
apiVersion: monitoring.coreos.com/v1alpha1 kind: Prometheus metadata: name: prometheus-frontend labels: prometheus: frontend spec: version: v1.3.0 #定義應(yīng)包括標(biāo)簽為“tier=frontend”的所有ServiceMonitor 到服務(wù)器的配置中 serviceMonitors: - selector: matchLabels: tier: frontend
現(xiàn)在Prometheus
將會監(jiān)控每個帶有tier: frontend
label的服務(wù)。
4. helm安裝
先決條件:
- 部署了
Helm
準(zhǔn)備好動手操作:
helm repo add coreos https://s3-eu-west-1.amazonaws.com/coreos-charts/stable/ helm install coreos/prometheus-operator --name prometheus-operator --namespace monitoring
到目前為止,我們已經(jīng)在我們的集群中安裝了Prometheus Operator
的TPR
。
現(xiàn)在我們來部署Prometheus
,Alertmanager
和Grafana
。
TIP: 當(dāng)我使用一個龐大的Helm Charts
時,我更傾向于創(chuàng)建一個獨立的value.yaml
文件將包含我所有自定義的變更。這么做使我和同事為后期的變化和修改更容易。
helm install coreos/kube-prometheus --name kube-prometheus \ -f my_changes/prometheus.yaml \ -f my_changes/grafana.yaml \ -f my_changes/alertmanager.yaml
檢查一切是否運行正常
kubectl -n monitoring get po NAME READY STATUS RESTARTS AGE alertmanager-kube-prometheus-0 2/2 Running 0 1h kube-prometheus-exporter-kube-state-68dbb4f7c9-tr6rp 2/2 Running 0 1h kube-prometheus-exporter-node-bqcj4 1/1 Running 0 1h kube-prometheus-exporter-node-jmcq2 1/1 Running 0 1h kube-prometheus-exporter-node-qnzsn 1/1 Running 0 1h kube-prometheus-exporter-node-v4wn8 1/1 Running 0 1h kube-prometheus-exporter-node-x5226 1/1 Running 0 1h kube-prometheus-exporter-node-z996c 1/1 Running 0 1h kube-prometheus-grafana-54c96ffc77-tjl6g 2/2 Running 0 1h prometheus-kube-prometheus-0 2/2 Running 0 1h prometheus-operator-1591343780-5vb5q 1/1 Running 0 1h
訪問下Prometheus UI
看一下Targets
頁面:
kubectl -n monitoring port-forward prometheus-kube-prometheus-0 9090 Forwarding from 127.0.0.1:9090 -> 9090
瀏覽器展示如下:
5. yaml文件安裝
此安裝方法本人親測有效,用到的yaml文件都打包好了。解壓之后直接kubectl apply即可用。會自動監(jiān)控當(dāng)前集群的所有node節(jié)點和pod。只需更改yaml文件中需要用到的鏡像。我這里都推到了公司公網(wǎng)harbor倉庫。部分鏡像已經(jīng)打成tar包。直接docker load -i即可用。
kube-state.tar.gz
webhook-dingtalk.tar.gz
prometheus-adapter.tar.gz
5.1 安裝
#軟件包集成了node Exporter alertmanager grafana prometheus ingress 所有服務(wù)的配置,只需解壓到K8S master中。 [root@lecode-k8s-master monitor]# ll total 1820 -rw-r--r-- 1 root root 875 Mar 11 2022 alertmanager-alertmanager.yaml -rw-r--r-- 1 root root 515 Mar 11 2022 alertmanager-podDisruptionBudget.yaml -rw-r--r-- 1 root root 4337 Mar 11 2022 alertmanager-prometheusRule.yaml -rw-r--r-- 1 root root 1483 Mar 14 2022 alertmanager-secret.yaml -rw-r--r-- 1 root root 301 Mar 11 2022 alertmanager-serviceAccount.yaml -rw-r--r-- 1 root root 540 Mar 11 2022 alertmanager-serviceMonitor.yaml -rw-r--r-- 1 root root 614 Mar 11 2022 alertmanager-service.yaml drwxr-x--- 2 root root 4096 Oct 25 13:49 backsvc #這里是grafana的service配置。nodeport模式。用于外部訪問。選擇使用 -rw-r--r-- 1 root root 278 Mar 11 2022 blackbox-exporter-clusterRoleBinding.yaml -rw-r--r-- 1 root root 287 Mar 11 2022 blackbox-exporter-clusterRole.yaml -rw-r--r-- 1 root root 1392 Mar 11 2022 blackbox-exporter-configuration.yaml -rw-r--r-- 1 root root 3081 Mar 11 2022 blackbox-exporter-deployment.yaml -rw-r--r-- 1 root root 96 Mar 11 2022 blackbox-exporter-serviceAccount.yaml -rw-r--r-- 1 root root 680 Mar 11 2022 blackbox-exporter-serviceMonitor.yaml -rw-r--r-- 1 root root 540 Mar 11 2022 blackbox-exporter-service.yaml -rw-r--r-- 1 root root 2521 Oct 25 13:36 dingtalk-dep.yaml -rw-r--r-- 1 root root 721 Mar 11 2022 grafana-dashboardDatasources.yaml -rw-r--r-- 1 root root 1448347 Mar 11 2022 grafana-dashboardDefinitions.yaml -rw-r--r-- 1 root root 625 Mar 11 2022 grafana-dashboardSources.yaml -rw-r--r-- 1 root root 8098 Mar 11 2022 grafana-deployment.yaml -rw-r--r-- 1 root root 86 Mar 11 2022 grafana-serviceAccount.yaml -rw-r--r-- 1 root root 398 Mar 11 2022 grafana-serviceMonitor.yaml -rw-r--r-- 1 root root 468 Mar 30 2022 grafana-service.yaml drwxr-xr-x 2 root root 4096 Oct 25 13:32 ingress #這里ingress資源也是可以直接用,可以把Prometheus和grafana服務(wù)暴露在外部。 -rw-r--r-- 1 root root 2639 Mar 14 2022 kube-prometheus-prometheusRule.yaml -rw-r--r-- 1 root root 3380 Mar 14 2022 kube-prometheus-prometheusRule.yamlbak -rw-r--r-- 1 root root 63531 Mar 11 2022 kubernetes-prometheusRule.yaml -rw-r--r-- 1 root root 6912 Mar 11 2022 kubernetes-serviceMonitorApiserver.yaml -rw-r--r-- 1 root root 425 Mar 11 2022 kubernetes-serviceMonitorCoreDNS.yaml -rw-r--r-- 1 root root 6431 Mar 11 2022 kubernetes-serviceMonitorKubeControllerManager.yaml -rw-r--r-- 1 root root 7629 Mar 11 2022 kubernetes-serviceMonitorKubelet.yaml -rw-r--r-- 1 root root 530 Mar 11 2022 kubernetes-serviceMonitorKubeScheduler.yaml -rw-r--r-- 1 root root 464 Mar 11 2022 kube-state-metrics-clusterRoleBinding.yaml -rw-r--r-- 1 root root 1712 Mar 11 2022 kube-state-metrics-clusterRole.yaml -rw-r--r-- 1 root root 2934 Oct 25 13:40 kube-state-metrics-deployment.yaml -rw-r--r-- 1 root root 3082 Mar 11 2022 kube-state-metrics-prometheusRule.yaml -rw-r--r-- 1 root root 280 Mar 11 2022 kube-state-metrics-serviceAccount.yaml -rw-r--r-- 1 root root 1011 Mar 11 2022 kube-state-metrics-serviceMonitor.yaml -rw-r--r-- 1 root root 580 Mar 11 2022 kube-state-metrics-service.yaml -rw-r--r-- 1 root root 444 Mar 11 2022 node-exporter-clusterRoleBinding.yaml -rw-r--r-- 1 root root 461 Mar 11 2022 node-exporter-clusterRole.yaml -rw-r--r-- 1 root root 3047 Mar 11 2022 node-exporter-daemonset.yaml -rw-r--r-- 1 root root 14356 Apr 11 2022 node-exporter-prometheusRule.yaml -rw-r--r-- 1 root root 270 Mar 11 2022 node-exporter-serviceAccount.yaml -rw-r--r-- 1 root root 850 Mar 11 2022 node-exporter-serviceMonitor.yaml -rw-r--r-- 1 root root 492 Mar 11 2022 node-exporter-service.yaml -rw-r--r-- 1 root root 482 Mar 11 2022 prometheus-adapter-apiService.yaml -rw-r--r-- 1 root root 576 Mar 11 2022 prometheus-adapter-clusterRoleAggregatedMetricsReader.yaml -rw-r--r-- 1 root root 494 Mar 11 2022 prometheus-adapter-clusterRoleBindingDelegator.yaml -rw-r--r-- 1 root root 471 Mar 11 2022 prometheus-adapter-clusterRoleBinding.yaml -rw-r--r-- 1 root root 378 Mar 11 2022 prometheus-adapter-clusterRoleServerResources.yaml -rw-r--r-- 1 root root 409 Mar 11 2022 prometheus-adapter-clusterRole.yaml -rw-r--r-- 1 root root 2204 Mar 11 2022 prometheus-adapter-configMap.yaml -rw-r--r-- 1 root root 2530 Oct 25 13:39 prometheus-adapter-deployment.yaml -rw-r--r-- 1 root root 506 Mar 11 2022 prometheus-adapter-podDisruptionBudget.yaml -rw-r--r-- 1 root root 515 Mar 11 2022 prometheus-adapter-roleBindingAuthReader.yaml -rw-r--r-- 1 root root 287 Mar 11 2022 prometheus-adapter-serviceAccount.yaml -rw-r--r-- 1 root root 677 Mar 11 2022 prometheus-adapter-serviceMonitor.yaml -rw-r--r-- 1 root root 501 Mar 11 2022 prometheus-adapter-service.yaml -rw-r--r-- 1 root root 447 Mar 11 2022 prometheus-clusterRoleBinding.yaml -rw-r--r-- 1 root root 394 Mar 11 2022 prometheus-clusterRole.yaml -rw-r--r-- 1 root root 5000 Mar 11 2022 prometheus-operator-prometheusRule.yaml -rw-r--r-- 1 root root 715 Mar 11 2022 prometheus-operator-serviceMonitor.yaml -rw-r--r-- 1 root root 499 Mar 11 2022 prometheus-podDisruptionBudget.yaml -rw-r--r-- 1 root root 14021 Mar 11 2022 prometheus-prometheusRule.yaml -rw-r--r-- 1 root root 1184 Mar 11 2022 prometheus-prometheus.yaml -rw-r--r-- 1 root root 471 Mar 11 2022 prometheus-roleBindingConfig.yaml -rw-r--r-- 1 root root 1547 Mar 11 2022 prometheus-roleBindingSpecificNamespaces.yaml -rw-r--r-- 1 root root 366 Mar 11 2022 prometheus-roleConfig.yaml -rw-r--r-- 1 root root 2047 Mar 11 2022 prometheus-roleSpecificNamespaces.yaml -rw-r--r-- 1 root root 271 Mar 11 2022 prometheus-serviceAccount.yaml -rw-r--r-- 1 root root 531 Mar 11 2022 prometheus-serviceMonitor.yaml -rw-r--r-- 1 root root 558 Mar 11 2022 prometheus-service.yaml drw-r--r-- 2 root root 4096 Oct 24 12:31 setup #先apply setup目錄中的yaml文件。然后apply一級目錄下的yaml文件。backsvc中的grafana的service資源清單。根據(jù)情況調(diào)整為nodeport或ClusterIP。K8S集群會自動在每臺K8S節(jié)點部署node-exporter并收集數(shù)據(jù)。登錄grafana后初始賬號密碼為admin admin。添加dashboard即可監(jiān)控K8S集群 [root@lecode-k8s-master monitor]# cd setup/ [root@lecode-k8s-master setup]# kubectl apply -f . [root@lecode-k8s-master setup]# cd .. [root@lecode-k8s-master monitor]# kubectl apply -f . [root@lecode-k8s-master monitor]# kubectl get po -n monitoring NAME READY STATUS RESTARTS AGE alertmanager-main-0 2/2 Running 0 74m alertmanager-main-1 2/2 Running 0 74m alertmanager-main-2 2/2 Running 0 74m blackbox-exporter-6798fb5bb4-d9m7m 3/3 Running 0 74m grafana-64668d8465-x7x9z 1/1 Running 0 74m kube-state-metrics-569d89897b-hlqxj 3/3 Running 0 57m node-exporter-6vqxg 2/2 Running 0 74m node-exporter-7dxh6 2/2 Running 0 74m node-exporter-9j5xk 2/2 Running 0 74m node-exporter-ftrmn 2/2 Running 0 74m node-exporter-qszkn 2/2 Running 0 74m node-exporter-wjkgj 2/2 Running 0 74m prometheus-adapter-5dd78c75c6-h2jf7 1/1 Running 0 58m prometheus-adapter-5dd78c75c6-qpwzv 1/1 Running 0 58m prometheus-k8s-0 2/2 Running 0 74m prometheus-k8s-1 2/2 Running 0 74m prometheus-operator-75d9b475d9-mmzgs 2/2 Running 0 80m webhook-dingtalk-6ffc94b49-z9z6l 1/1 Running 0 61m [root@lecode-k8s-master backsvc]# kubectl get svc -n monitoring NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager-main NodePort 10.98.35.93 <none> 9093:30093/TCP 72m alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 72m blackbox-exporter ClusterIP 10.109.10.110 <none> 9115/TCP,19115/TCP 72m grafana NodePort 10.110.48.214 <none> 3000:30300/TCP 72m kube-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 72m node-exporter ClusterIP None <none> 9100/TCP 72m prometheus-adapter ClusterIP 10.97.23.176 <none> 443/TCP 72m prometheus-k8s ClusterIP 10.100.92.254 <none> 9090/TCP 72m prometheus-operated ClusterIP None <none> 9090/TCP 72m prometheus-operator ClusterIP None <none> 8443/TCP 78m webhook-dingtalk ClusterIP 10.100.131.63 <none> 80/TCP 72m
5.2 訪問服務(wù)
暴露服務(wù)三種方法:用service資源的nodeport模式,或者用k8s的ingress暴露服務(wù)或者本地nginx代理。本地的nginx代理模式
這里我grafana用的是nodeport模式。Prometheus用的是nginx代理。
附上nginx配置文件
[root@lecode-k8s-master setup]# cat /usr/local/nginx/conf/4-layer-conf.d/lecode-prometheus-operator.conf #代理prometheus內(nèi)置Dashboard UI upstream prometheus-dashboard { server 10.100.92.254:9090; #這里ip為prometheus-k8s svc資源的ip } server { listen 9090; proxy_pass prometheus-dashboard; } #代理grafana upstream grafana { server 10.1.82.89:3000; #這里ip為grafana svc資源的ip } server { listen 3000; proxy_pass grafana; }
訪問Prometheus targets
5.3 接入grafana
訪問grafana(默認(rèn)密碼是admin admin)
去grafana官網(wǎng)下載對應(yīng)dashboard 地址:https://grafana.com/grafana/dashboards/
6. 監(jiān)控集群外的服務(wù)
6.1 exporter安裝
在對應(yīng)服務(wù)的本地安裝對應(yīng)的exporter用于收集數(shù)據(jù)(這里以mysql為例)
#下載對應(yīng)服務(wù)的exporter #插件下載地址:https://www.modb.pro/db/216588 #插件下載地址:https://prometheus.io/download/ #下載完成后解壓mysqld_exporter-0.13.0.linux-amd64.tar.gz #配置mysql-exporter 在root路徑下,創(chuàng)建.my.cnf文件。內(nèi)容如下: [root@lecode-test-001 ~]# cat /root/.my.cnf [client] user=mysql_monitor password=Mysql@123 #創(chuàng)建mysql 用戶并授權(quán) CREATE USER ‘mysql_monitor'@‘localhost' IDENTIFIED BY ‘Mysql@123' WITH MAX_USER_CONNECTIONS 3; GRANT PROCESS, REPLICATION CLIENT, SELECT ON . TO ‘mysql_monitor'@‘localhost'; FLUSH PRIVILEGES; EXIT #啟動mysqld_exporter [root@lecode-test-001 mysql-exporter]# nohup mysqld_exporter & #找到對應(yīng)的端口 [root@lecode-test-001 mysql-exporter]# tail -f nohup.out level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:277 msg="Starting msqyld_exporter" version="(version=0.13.0, branch=HEAD, revision=ad2847c7fa67b9debafccd5a08bacb12fc9031f1)" level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:278 msg="Build context" (gogo1.16.4,userroot@e2043849cb1f,date20210531-07:30:16)=(MISSING) level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=global_status level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=global_variables level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=slave_status level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=info_schema.innodb_cmp level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=info_schema.innodb_cmpmem level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=info_schema.query_response_time level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:303 msg="Listening on address" address=:9104 #這是exporter的端口 level=info ts=2022-10-25T09:26:54.464Z caller=tls_config.go:191 msg="TLS is disabled." http2=false #檢查端口 [root@lecode-test-001 mysql-exporter]# ss -lntup |grep 9104 tcp LISTEN 0 128 :::9104 :::* users:(("mysqld_exporter",pid=26115,fd=3))
6.2 K8S配置
創(chuàng)建endpoint資源關(guān)聯(lián)對應(yīng)服務(wù)主機的exporter端口。綁定service資源,通過ServiceMonitor資源添加Prometheus targets,
1)官方格式
kubectl -n monitoring get prometheus kube-prometheus -o yaml apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: labels: app: prometheus chart: prometheus-0.0.14 heritage: Tiller prometheus: kube-prometheus release: kube-prometheus name: kube-prometheus namespace: monitoring spec: ... baseImage: quay.io/prometheus/prometheus serviceMonitorSelector: matchLabels: prometheus: kube-prometheus #接下來就是按照格式創(chuàng)建對應(yīng)的ServiceMonitor資源
通過ep資源把外部服務(wù)關(guān)聯(lián)到K8S內(nèi)部,綁定對應(yīng)的svc資源。在由serviceMonitor綁定對應(yīng)的svc資源把數(shù)據(jù)提交給Prometheus,serviceMonitor通過標(biāo)簽選擇器關(guān)聯(lián)service,而service只需要通過端口關(guān)聯(lián)ep,這里的標(biāo)簽和端口一定要注意一致.
2) 創(chuàng)建資源
[root@lecode-k8s-master monitor]# cat mysql.yaml apiVersion: v1 kind: Endpoints metadata: name: mysql-test namespace: monitoring subsets: - addresses: - ip: 192.168.1.17 # ip為安裝應(yīng)用服務(wù)器的ip ports: - name: mysql port: 9104 #export的端口 protocol: TCP --- apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/component: mysql app.kubernetes.io/name: mysql app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 0.49.0 name: mysql-test namespace: monitoring spec: clusterIP: None clusterIPs: - None ports: - name: mysql port: 9104 protocol: TCP sessionAffinity: None type: ClusterIP --- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: app.kubernetes.io/component: mysql app.kubernetes.io/name: mysql app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 0.49.0 name: mysql-test namespace: monitoring spec: endpoints: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token interval: 30s port: mysql tlsConfig: insecureSkipVerify: true selector: matchLabels: app.kubernetes.io/component: mysql app.kubernetes.io/name: mysql app.kubernetes.io/part-of: kube-prometheus #創(chuàng)建 [root@lecode-k8s-master monitor]# kubectl apply -f mysql.yaml endpoints/mysql-test created service/mysql-test created servicemonitor.monitoring.coreos.com/mysql-test created #檢查 [root@lecode-k8s-master monitor]# kubectl get -f mysql.yaml NAME ENDPOINTS AGE endpoints/mysql-test 192.168.1.17:9104 10m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/mysql-test ClusterIP None <none> 9104/TCP 10m NAME AGE servicemonitor.monitoring.coreos.com/mysql-test 10m
6.3 接入grafana
7 監(jiān)控集群外的服務(wù)-redis
7.1 exporter安裝
#部署redis-exporter #插件下載地址:https://www.modb.pro/db/216588 [root@lecode-test-001 ~]# tar xf redis_exporter-v1.3.2.linux-amd64.tar.gz [root@lecode-test-001 ~]# ll drwxr-xr-x 2 root root 4096 Nov 6 2019 redis_exporter-v1.3.2.linux-amd64 -rw-r--r-- 1 root root 3376155 Oct 27 10:26 redis_exporter-v1.3.2.linux-amd64.tar.gz [root@lecode-test-001 ~]# mv redis_exporter-v1.3.2.linux-amd64 redis_exporter [root@lecode-test-001 ~]# cd redis_exporter [root@lecode-test-001 redis_exporter]# ll total 8488 -rw-r--r-- 1 root root 1063 Nov 6 2019 LICENSE -rw-r--r-- 1 root root 10284 Nov 6 2019 README.md -rwxr-xr-x 1 root root 8675328 Nov 6 2019 redis_exporter [root@lecode-test-001 redis_exporter]# nohup ./redis_exporter -redis.addr 192.168.1.17:6379 -redis.password 'Redislecodetest@shuli123' & [1] 4564 [root@lecode-test-001 redis_exporter]# nohup: ignoring input and appending output to a€?nohup.outa€? [root@lecode-test-001 redis_exporter]# tail -f nohup.out time="2022-10-27T10:26:48+08:00" level=info msg="Redis Metrics Exporter v1.3.2 build date: 2019-11-06-02:25:20 sha1: 175a69f33e8267e0a0ba47caab488db5e83a592e Go: go1.13.4 GOOS: linux GOARCH: amd64" time="2022-10-27T10:26:48+08:00" level=info msg="Providing metrics at :9121/metrics" #端口為9121
7.2 K8S配置
#創(chuàng)建redis-serviceMonitor資源 [root@lecode-k8s-master monitor]# cat redis.yaml apiVersion: v1 kind: Endpoints metadata: name: redis-test namespace: monitoring subsets: - addresses: - ip: 192.168.1.17 # ip為安裝應(yīng)用服務(wù)器的ip ports: - name: redis port: 9121 #exporter端口 protocol: TCP --- apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/component: redis app.kubernetes.io/name: redis app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 0.49.0 name: redis-test namespace: monitoring spec: clusterIP: None clusterIPs: - None ports: - name: redis port: 9121 protocol: TCP sessionAffinity: None type: ClusterIP --- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: app.kubernetes.io/component: redis app.kubernetes.io/name: redis app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 0.49.0 name: redis-test namespace: monitoring spec: endpoints: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token interval: 30s port: redis tlsConfig: insecureSkipVerify: true selector: matchLabels: app.kubernetes.io/component: redis app.kubernetes.io/name: redis app.kubernetes.io/part-of: kube-prometheus #創(chuàng)建資源 [root@lecode-k8s-master monitor]# kubectl apply -f redis.yaml endpoints/redis-test created service/redis-test created servicemonitor.monitoring.coreos.com/redis-test created [root@lecode-k8s-master monitor]# kubectl get ep,svc,serviceMonitor -n monitoring |grep redis endpoints/redis-test 192.168.1.17:9121 6m2s service/redis-test ClusterIP None <none> 9121/TCP 6m2s servicemonitor.monitoring.coreos.com/redis-test 6m2s
7.3 Prometheus檢查
7.4 接入grafana
dashboard ID: 11835
8. 監(jiān)控集群外的服務(wù)-kafka
8.1 安裝exporter
#插件下載地址:https://www.modb.pro/db/216588 [root@lecode-test-001 ~]# tar xf kafka_exporter-1.4.2.linux-amd64.tar.gz [root@lecode-test-001 ~]# ll drwxrwxr-x 2 2000 2000 4096 Sep 16 2021 kafka_exporter-1.4.2.linux-amd64 -rw-r--r-- 1 root root 8499720 Oct 27 15:30 kafka_exporter-1.4.2.linux-amd64.tar.gz [root@lecode-test-001 ~]# mv kafka_exporter-1.4.2.linux-amd64 kafka_exporter [root@lecode-test-001 ~]# cd kafka_exporter [root@lecode-test-001 kafka_exporter]# ll total 17676 -rwxr-xr-x 1 2000 2000 18086208 Sep 16 2021 kafka_exporter -rw-rw-r-- 1 2000 2000 11357 Sep 16 2021 LICENSE [root@lecode-test-001 kafka_exporter]# nohup ./kafka_exporter --kafka.server=192.168.1.17:9092 & [1] 20777 [root@lecode-test-001 kafka_exporter]# nohup: ignoring input and appending output to ‘nohup.out' [root@lecode-test-001 kafka_exporter]# tail -f nohup.out I1027 15:32:38.904075 20777 kafka_exporter.go:769] Starting kafka_exporter (version=1.4.2, branch=HEAD, revision=0d5d4ac4ba63948748cc2c53b35ed95c310cd6f2) I1027 15:32:38.905515 20777 kafka_exporter.go:929] Listening on HTTP :9308 #exporter端口為9308
8.2 K8S配置
[root@lecode-k8s-master monitor]# cat kafka.yaml apiVersion: v1 kind: Endpoints metadata: name: kafka-test namespace: monitoring subsets: - addresses: - ip: 192.168.1.17 # ip為安裝應(yīng)用服務(wù)器的ip ports: - name: kafka port: 9308 #export的端口 protocol: TCP --- apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/component: kafka app.kubernetes.io/name: kafka app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 0.49.0 name: kafka-test namespace: monitoring spec: clusterIP: None clusterIPs: - None ports: - name: kafka port: 9308 protocol: TCP sessionAffinity: None type: ClusterIP --- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: app.kubernetes.io/component: kafka app.kubernetes.io/name: kafka app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 0.49.0 name: kafka-test namespace: monitoring spec: endpoints: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token interval: 30s port: kafka tlsConfig: insecureSkipVerify: true selector: matchLabels: app.kubernetes.io/component: kafka app.kubernetes.io/name: kafka app.kubernetes.io/part-of: kube-prometheus #創(chuàng)建 [root@lecode-k8s-master monitor]# kubectl apply -f kafka.yaml endpoints/kafka-test created service/kafka-test created servicemonitor.monitoring.coreos.com/kafka-test created [root@lecode-k8s-master monitor]# kubectl get -f kafka.yaml NAME ENDPOINTS AGE endpoints/kafka-test 192.168.1.17:9308 8m49s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kafka-test ClusterIP None <none> 9308/TCP 8m49s NAME AGE servicemonitor.monitoring.coreos.com/kafka-test 8m48s
8.3 Prometheus檢查
8.4 接入grafana
dashboard ID:7589
9. 監(jiān)控集群外的服務(wù)-zookeeper
9.1 安裝exporter
exporter下載地址:https://github.com/carlpett/zookeeper_exporter/releases/download/v1.0.2/zookeeper_exporter [root@lecode-test-001 zookeeper_exporter]# nohup ./zookeeper_exporter -zookeeper 192.168.1.17:2181 -bind-addr :9143 & [2] 8310 [root@lecode-test-001 zookeeper_exporter]# nohup: ignoring input and appending output to ‘nohup.out' [root@lecode-test-001 zookeeper_exporter]# tail -f nohup.out time="2022-10-27T15:58:27+08:00" level=info msg="zookeeper_exporter, version v1.0.2 (branch: HEAD, revision: d6e929223f6b3bf5ff25dd0340e8194cbd4d04fc)\n build user: @bd731f434d23\n build date: 2018-05-01T20:40:14+0000\n go version: go1.10.1" time="2022-10-27T15:58:27+08:00" level=info msg="Starting zookeeper_exporter" time="2022-10-27T15:58:27+08:00" level=info msg="Starting metric http endpoint on :9143" #exporter端口為9143
9.2 K8S配置
[root@lecode-k8s-master monitor]# cat zookeeper.yaml apiVersion: v1 kind: Endpoints metadata: name: zookeeper-test namespace: monitoring subsets: - addresses: - ip: 192.168.1.17 ports: - name: zookeeper port: 9143 protocol: TCP --- apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/component: zookeeper app.kubernetes.io/name: zookeeper app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 0.49.0 name: zookeeper-test namespace: monitoring spec: clusterIP: None clusterIPs: - None ports: - name: zookeeper port: 9143 protocol: TCP sessionAffinity: None type: ClusterIP --- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: app.kubernetes.io/component: zookeeper app.kubernetes.io/name: zookeeper app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 0.49.0 name: zookeeper-test namespace: monitoring spec: endpoints: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token interval: 30s port: zookeeper tlsConfig: insecureSkipVerify: true selector: matchLabels: app.kubernetes.io/component: zookeeper app.kubernetes.io/name: zookeeper app.kubernetes.io/part-of: kube-prometheus #創(chuàng)建 [root@lecode-k8s-master monitor]# kubectl apply -f zookeeper.yaml endpoints/zookeeper-test created service/zookeeper-test created servicemonitor.monitoring.coreos.com/zookeeper-test created [root@lecode-k8s-master monitor]# kubectl get -f zookeeper.yaml NAME ENDPOINTS AGE endpoints/zookeeper-test 192.168.1.17:9143 9m55s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/zookeeper-test ClusterIP None <none> 9143/TCP 9m55s NAME AGE servicemonitor.monitoring.coreos.com/zookeeper-test 9m55s
9.3 Prometheus檢查
9.4 接入grafana
dashboard ID:15026
以上就是K8S prometheus operator監(jiān)控工作原理介紹的詳細內(nèi)容,更多關(guān)于K8S prometheus operator監(jiān)控的資料請關(guān)注腳本之家其它相關(guān)文章!
相關(guān)文章
Kubernetes(k8s?1.23))安裝與卸載詳細教程
這篇文章主要介紹了Kubernetes(k8s?1.23))安裝與卸載,本文給大家介紹的非常詳細,對大家的學(xué)習(xí)或工作具有一定的參考借鑒價值,需要的朋友可以參考下2022-07-07k8s跨服務(wù)調(diào)用入門到實戰(zhàn)示例詳解
這篇文章主要為大家介紹了k8s跨服務(wù)調(diào)用入門到實戰(zhàn)示例詳解,有需要的朋友可以借鑒參考下,希望能夠有所幫助,祝大家多多進步,早日升職加薪2023-09-09詳解kubelet?創(chuàng)建pod流程代碼圖解及日志說明
這篇文章主要為大家介紹了詳解kubelet?創(chuàng)建pod流程代碼圖解及日志說明,有需要的朋友可以借鑒參考下,希望能夠有所幫助,祝大家多多進步,早日升職加薪2022-09-09