K8S部署rocketmq5全過(guò)程
更新時(shí)間：2025年01月02日 09:22:06 作者：萊特昂
在開(kāi)發(fā)環(huán)境中部署RocketMQ 5并驗(yàn)證新版本代理特性,遇到一系列問(wèn)題,包括存儲(chǔ)權(quán)限、主從副本配置和命名空間問(wèn)題,通過(guò)修改配置文件和PV設(shè)置,最終解決了這些問(wèn)題,成功部署了RocketMQ集群
背景
需要在開(kāi)發(fā)環(huán)境部署rocketmq5驗(yàn)證新版本proxy相關(guān)的特性，而開(kāi)發(fā)環(huán)境沒(méi)有helm和外網(wǎng)，有k8s的環(huán)境
發(fā)現(xiàn)網(wǎng)上也沒(méi)有太多資料，記錄一下操作流程
操作流程
1. helm庫(kù)拉取rocketmq5鏡像
用的是某個(gè)大佬上傳的helm庫(kù)鏡像：
## 添加 helm 倉(cāng)庫(kù) helm repo add rocketmq-repo https://helm-charts.itboon.top/rocketmq helm repo update rocketmq-repo ## 查看鏡像 helm search rocketmq 拉取鏡像到本地 兩個(gè)都拉 helm pull itboon/rocketmq helm pull itboon/rocketmq-cluster 解壓 tar -zxf rocketmq.tgz
2. 單集群?jiǎn)?dòng)測(cè)試
進(jìn)入目錄修改value.yaml文件：
clusterName: "rocketmq-helm"
image:
repository: "apache/rocketmq"
pullPolicy: IfNotPresent
tag: "5.3.0"
podSecurityContext:
fsGroup: 3000
runAsUser: 3000
broker:
size:
master: 1
replica: 0
# podSecurityContext: {}
# containerSecurityContext: {}
master:
brokerRole: ASYNC_MASTER
jvm:
maxHeapSize: 256M
# javaOptsOverride: ""
resources:
limits:
cpu: 2
memory: 512Mi
requests:
cpu: 200m
memory: 256Mi
replica:
jvm:
maxHeapSize: 256M
# javaOptsOverride: ""
resources:
limits:
cpu: 4
memory: 512Mi
requests:
cpu: 50m
memory: 256Mi
hostNetwork: false
persistence:
enabled: true
size: 100Mi
#storageClass: "local-storage"
aclConfigMapEnabled: false
aclConfig: |
globalWhiteRemoteAddresses:
- '*'
- 10.*.*.*
- 192.168.*.*
config:
## brokerClusterName brokerName brokerRole brokerId 由內(nèi)置腳本自動(dòng)生成
deleteWhen: "04"
fileReservedTime: "48"
flushDiskType: "ASYNC_FLUSH"
waitTimeMillsInSendQueue: "1000"
# aclEnable: true
affinityOverride: {}
tolerations: []
nodeSelector: {}
## broker.readinessProbe
readinessProbe:
tcpSocket:
port: main
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 6
nameserver:
replicaCount: 1
jvm:
maxHeapSize: 256M
# javaOptsOverride: ""
resources:
limits:
cpu: 2
memory: 256Mi
ephemeral-storage: 256Mi
requests:
cpu: 100m
memory: 256Mi
ephemeral-storage: 256Mi
persistence:
enabled: false
size: 256Mi
#storageClass: "local-storage"
affinityOverride: {}
tolerations: []
nodeSelector: {}
## nameserver.readinessProbe
readinessProbe:
tcpSocket:
port: main
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 6
## nameserver.service
service:
annotations: {}
type: ClusterIP
proxy:
enabled: true
replicaCount: 1
jvm:
maxHeapSize: 600M
# javaOptsOverride: ""
resources:
limits:
cpu: 2
memory: 512Mi
requests:
cpu: 100m
memory: 256Mi
affinityOverride: {}
tolerations: []
nodeSelector: {}
## proxy.readinessProbe
readinessProbe:
tcpSocket:
port: main
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 6
## proxy.service
service:
annotations: {}
type: ClusterIP
dashboard:
enabled: true
replicaCount: 1
image:
repository: "apacherocketmq/rocketmq-dashboard"
pullPolicy: IfNotPresent
tag: "1.0.0"
auth:
enabled: true
users:
- name: admin
password: admin
isAdmin: true
- name: user01
password: userPass
jvm:
maxHeapSize: 256M
resources:
limits:
cpu: 1
memory: 512Mi
requests:
cpu: 20m
memory: 512Mi
## dashboard.readinessProbe
readinessProbe:
failureThreshold: 6
httpGet:
path: /
port: http
livenessProbe: {}
service:
annotations: {}
type: ClusterIP
# nodePort: 31007
ingress:
enabled: false
className: ""
annotations: {}
# nginx.ingress.kubernetes.io/whitelist-source-range: 10.0.0.0/8,124.160.30.50
hosts:
- host: rocketmq-dashboard.example.com
tls: []
# - secretName: example-tls
# hosts:
# - rocketmq-dashboard.example.com
## controller mode is an experimental feature
controllerModeEnabled: false
controller:
enabled: false
jvm:
maxHeapSize: 256M
# javaOptsOverride: ""
resources:
limits:
cpu: 2
memory: 512Mi
requests:
cpu: 100m
memory: 256Mi
persistence:
enabled: true
size: 256Mi
accessModes:
- ReadWriteOnce
## controller.service
service:
annotations: {}
## controller.config
config:
controllerDLegerGroup: group1
enableElectUncleanMaster: false
notifyBrokerRoleChanged: true
## controller.readinessProbe
readinessProbe:
tcpSocket:
port: main
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 6helm啟動(dòng)
helm upgrade --install rocketmq \ --namespace rocketmq-demo \ --create-namespace \ --set broker.persistence.enabled="false" \ ./rocketmq
3. sc/pv配置
采用的掛載本地的方式設(shè)置：
SC:
vi sc_local.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
annotations:
openebs.io/cas-type: local
storageclass.kubernetes.io/is-default-class: "false"
cas.openebs.io/config: |
#hostpath type will create a PV by
# creating a sub-directory under the
# BASEPATH provided below.
- name: StorageType
value: "hostpath"
#Specify the location (directory) where
# where PV(volume) data will be saved.
# A sub-directory with pv-name will be
# created. When the volume is deleted,
# the PV sub-directory will be deleted.
#Default value is /var/openebs/local
- name: BasePath
value: "/tmp/storage"
provisioner: openebs.io/local
volumeBindingMode: Immediate
reclaimPolicy: Retain
kubectl apply -f sc_local.yamlPV（只broker）：
vi local_pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
labels:
type: local
name: broker-storage-rocketmq-broker-master-0
namespace: rocketmq-demo
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 100Mi
hostPath:
path: /tmp/storage
persistentVolumeReclaimPolicy: Recycle
storageClassName: local-storage
volumeMode: Filesystem
---
apiVersion: v1
kind: PersistentVolume
metadata:
labels:
type: local
name: broker-storage-rocketmq-broker-replica-id1-0
namespace: rocketmq-demo
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 100Mi
hostPath:
path: /tmp/storageSlave
persistentVolumeReclaimPolicy: Recycle
storageClassName: local-storage
volumeMode: Filesystem
kubectl apply -f local_pv.yaml
kubectl delete pv --all 4.集群?jiǎn)?dòng)測(cè)試
修改value.yaml，主要降低了配置：
clusterName: "rocketmq-helm"
nameOverride: rocketmq
image:
repository: "apache/rocketmq"
pullPolicy: IfNotPresent
tag: "5.3.0"
podSecurityContext:
fsGroup: 3000
runAsUser: 3000
broker:
size:
master: 1
replica: 1
# podSecurityContext: {}
# containerSecurityContext: {}
master:
brokerRole: ASYNC_MASTER
jvm:
maxHeapSize: 512M
# javaOptsOverride: ""
resources:
limits:
cpu: 2
memory: 512Mi
requests:
cpu: 100m
memory: 128Mi
replica:
jvm:
maxHeapSize: 256M
# javaOptsOverride: ""
resources:
limits:
cpu: 2
memory: 256Mi
requests:
cpu: 50m
memory: 128Mi
hostNetwork: false
persistence:
enabled: true
size: 100Mi
#storageClass: "local-storage"
aclConfigMapEnabled: false
aclConfig: |
globalWhiteRemoteAddresses:
- '*'
- 10.*.*.*
- 192.168.*.*
config:
## brokerClusterName brokerName brokerRole brokerId 由內(nèi)置腳本自動(dòng)生成
deleteWhen: "04"
fileReservedTime: "48"
flushDiskType: "ASYNC_FLUSH"
waitTimeMillsInSendQueue: "1000"
# aclEnable: true
affinityOverride: {}
tolerations: []
nodeSelector: {}
## broker.readinessProbe
readinessProbe:
tcpSocket:
port: main
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 6
nameserver:
replicaCount: 1
jvm:
maxHeapSize: 256M
# javaOptsOverride: ""
resources:
limits:
cpu: 1
memory: 256Mi
ephemeral-storage: 256Mi
requests:
cpu: 100m
memory: 128Mi
ephemeral-storage: 128Mi
persistence:
enabled: false
size: 128Mi
#storageClass: "local-storage"
affinityOverride: {}
tolerations: []
nodeSelector: {}
## nameserver.readinessProbe
readinessProbe:
tcpSocket:
port: main
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 6
## nameserver.service
service:
annotations: {}
type: ClusterIP
proxy:
enabled: true
replicaCount: 2
jvm:
maxHeapSize: 512M
# javaOptsOverride: ""
resources:
limits:
cpu: 2
memory: 512Mi
requests:
cpu: 100m
memory: 256Mi
affinityOverride: {}
tolerations: []
nodeSelector: {}
## proxy.readinessProbe
readinessProbe:
tcpSocket:
port: main
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 6
## proxy.service
service:
annotations: {}
type: ClusterIP
dashboard:
enabled: false
replicaCount: 1
image:
repository: "apacherocketmq/rocketmq-dashboard"
pullPolicy: IfNotPresent
tag: "1.0.0"
auth:
enabled: true
users:
- name: admin
password: admin
isAdmin: true
- name: user01
password: userPass
jvm:
maxHeapSize: 256M
resources:
limits:
cpu: 1
memory: 256Mi
requests:
cpu: 20m
memory: 128Mi
## dashboard.readinessProbe
readinessProbe:
failureThreshold: 6
httpGet:
path: /
port: http
livenessProbe: {}
service:
annotations: {}
type: ClusterIP
# nodePort: 31007
ingress:
enabled: false
className: ""
annotations: {}
# nginx.ingress.kubernetes.io/whitelist-source-range: 10.0.0.0/8,124.160.30.50
hosts:
- host: rocketmq-dashboard.example.com
tls: []
# - secretName: example-tls
# hosts:
# - rocketmq-dashboard.example.com
## controller mode is an experimental feature
controllerModeEnabled: false
controller:
enabled: false
replicaCount: 3
jvm:
maxHeapSize: 256M
# javaOptsOverride: ""
resources:
limits:
cpu: 2
memory: 256Mi
requests:
cpu: 100m
memory: 128Mi
persistence:
enabled: true
size: 128Mi
accessModes:
- ReadWriteOnce
## controller.service
service:
annotations: {}
## controller.config
config:
controllerDLegerGroup: group1
enableElectUncleanMaster: false
notifyBrokerRoleChanged: true
## controller.readinessProbe
readinessProbe:
tcpSocket:
port: main
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 65.離線(xiàn)安裝
helm導(dǎo)出yaml文件：
helm template rocketmq ./rocketmq-cluster --output-dir ./rocketmq-cluster-yaml
注意，轉(zhuǎn)成yaml文件后，原本用helm設(shè)置的namespace沒(méi)了。
執(zhí)行yaml文件驗(yàn)證：
kubectl apply -f rocketmq-cluster-yaml/ --recursive kubectl delete -f rocketmq-cluster-yaml/ --recursive
yaml導(dǎo)出：
## 安裝傳輸工具 yum install lrzsz ## 打包yaml文件夾 tar czvf folder.tar.gz itboon sz folder.tar.gz
附錄
最后生成的部署yaml：
- nameserver
---
# Source: rocketmq-cluster/templates/nameserver/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: "rocketmq-nameserver"
namespace: rocketmq
labels:
helm.sh/chart: rocketmq-cluster-12.3.2
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
app.kubernetes.io/version: "5.3.0"
app.kubernetes.io/managed-by: Helm
spec:
minReadySeconds: 20
replicas: 1
podManagementPolicy: Parallel
selector:
matchLabels:
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
component: nameserver
serviceName: "rocketmq-nameserver-headless"
template:
metadata:
annotations:
checksum/config: 9323bc706d85f980c210e9823264a63548598b649c4935f9db6559d4fecbcc93
labels:
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
component: nameserver
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 5
podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
component: nameserver
topologyKey: kubernetes.io/hostname
securityContext:
fsGroup: 3000
runAsUser: 3000
containers:
- name: nameserver
image: "apache/rocketmq:5.3.0"
imagePullPolicy: IfNotPresent
command:
- sh
- /mq-server-start.sh
env:
- name: ROCKETMQ_PROCESS_ROLE
value: nameserver
- name: ROCKETMQ_JAVA_OPTIONS_HEAP
value: -Xms512M -Xmx512M
ports:
- containerPort: 9876
name: main
protocol: TCP
resources:
limits:
cpu: 1
ephemeral-storage: 512Mi
memory: 512Mi
requests:
cpu: 100m
ephemeral-storage: 256Mi
memory: 256Mi
readinessProbe:
failureThreshold: 6
initialDelaySeconds: 10
periodSeconds: 10
tcpSocket:
port: main
timeoutSeconds: 3
lifecycle:
preStop:
exec:
command: ["sh", "-c", "sleep 5; ./mqshutdown namesrv"]
volumeMounts:
- mountPath: /mq-server-start.sh
name: mq-server-start-sh
subPath: mq-server-start.sh
- mountPath: /etc/rocketmq/base-cm
name: base-cm
- mountPath: /home/rocketmq/logs
name: nameserver-storage
subPath: logs
dnsPolicy: ClusterFirst
terminationGracePeriodSeconds: 15
volumes:
- configMap:
items:
- key: mq-server-start.sh
path: mq-server-start.sh
name: rocketmq-server-config
defaultMode: 0755
name: mq-server-start-sh
- configMap:
name: rocketmq-server-config
name: base-cm
- name: nameserver-storage
emptyDir: {}
---
# Source: rocketmq-cluster/templates/nameserver/svc.yaml
apiVersion: v1
kind: Service
metadata:
name: rocketmq-nameserver
labels:
helm.sh/chart: rocketmq-cluster-12.3.2
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
app.kubernetes.io/version: "5.3.0"
app.kubernetes.io/managed-by: Helm
component: nameserver
spec:
ports:
- port: 9876
protocol: TCP
targetPort: 9876
selector:
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
component: nameserver
type: "ClusterIP"
---
# Source: rocketmq-cluster/templates/nameserver/svc-headless.yaml
apiVersion: v1
kind: Service
metadata:
name: "rocketmq-nameserver-headless"
labels:
helm.sh/chart: rocketmq-cluster-12.3.2
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
app.kubernetes.io/version: "5.3.0"
app.kubernetes.io/managed-by: Helm
component: nameserver
spec:
clusterIP: "None"
publishNotReadyAddresses: true
ports:
- port: 9876
protocol: TCP
targetPort: 9876
selector:
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
component: nameserver
- broker
---
# Source: rocketmq-cluster/templates/broker/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: rocketmq-broker-master
namespace: rocketmq
labels:
helm.sh/chart: rocketmq-cluster-12.3.2
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
app.kubernetes.io/version: "5.3.0"
app.kubernetes.io/managed-by: Helm
spec:
minReadySeconds: 20
replicas: 1
podManagementPolicy: OrderedReady
selector:
matchLabels:
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
component: broker
broker: rocketmq-broker-master
serviceName: ""
template:
metadata:
annotations:
checksum/config: 9323bc706d85f980c210e9823264a63548598b649c4935f9db6559d4fecbcc93
labels:
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
component: broker
broker: rocketmq-broker-master
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 5
podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
component: broker
topologyKey: kubernetes.io/hostname
securityContext:
fsGroup: 3000
runAsUser: 3000
containers:
- name: broker
image: "apache/rocketmq:5.3.0"
imagePullPolicy: IfNotPresent
command:
- sh
- /mq-server-start.sh
env:
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: ROCKETMQ_PROCESS_ROLE
value: broker
- name: NAMESRV_ADDR
value: rocketmq-nameserver-0.rocketmq-nameserver-headless.rocketmq.svc:9876
- name: ROCKETMQ_CONF_brokerId
value: "0"
- name: ROCKETMQ_CONF_brokerRole
value: "ASYNC_MASTER"
- name: ROCKETMQ_JAVA_OPTIONS_HEAP
value: -Xms1G -Xmx1G
ports:
- containerPort: 10909
name: vip
protocol: TCP
- containerPort: 10911
name: main
protocol: TCP
- containerPort: 10912
name: ha
protocol: TCP
resources:
limits:
cpu: 2
memory: 2Gi
requests:
cpu: 100m
memory: 512Mi
readinessProbe:
failureThreshold: 6
initialDelaySeconds: 10
periodSeconds: 10
tcpSocket:
port: main
timeoutSeconds: 3
lifecycle:
preStop:
exec:
command: ["sh", "-c", "sleep 5; ./mqshutdown broker"]
volumeMounts:
- mountPath: /home/rocketmq/logs
name: broker-storage
subPath: rocketmq-broker/logs
- mountPath: /home/rocketmq/store
name: broker-storage
subPath: rocketmq-broker/store
- mountPath: /etc/rocketmq/broker-base.conf
name: broker-base-config
subPath: broker-base.conf
- mountPath: /mq-server-start.sh
name: mq-server-start-sh
subPath: mq-server-start.sh
dnsPolicy: ClusterFirst
terminationGracePeriodSeconds: 30
volumes:
- configMap:
items:
- key: broker-base.conf
path: broker-base.conf
name: rocketmq-server-config
name: broker-base-config
- configMap:
items:
- key: mq-server-start.sh
path: mq-server-start.sh
name: rocketmq-server-config
defaultMode: 0755
name: mq-server-start-sh
volumeClaimTemplates:
- metadata:
name: broker-storage
spec:
accessModes:
- ReadWriteOnce
storageClassName: local-path
resources:
requests:
storage: "100Mi"
---
# Source: rocketmq-cluster/templates/broker/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: rocketmq-broker-replica-id1
namespace: rocketmq
labels:
helm.sh/chart: rocketmq-cluster-12.3.2
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
app.kubernetes.io/version: "5.3.0"
app.kubernetes.io/managed-by: Helm
spec:
minReadySeconds: 20
replicas: 1
podManagementPolicy: OrderedReady
selector:
matchLabels:
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
component: broker
broker: rocketmq-broker-replica-id1
serviceName: ""
template:
metadata:
annotations:
checksum/config: 9323bc706d85f980c210e9823264a63548598b649c4935f9db6559d4fecbcc93
labels:
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
component: broker
broker: rocketmq-broker-replica-id1
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 5
podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
component: broker
topologyKey: kubernetes.io/hostname
securityContext:
fsGroup: 3000
runAsUser: 3000
containers:
- name: broker
image: "apache/rocketmq:5.3.0"
imagePullPolicy: IfNotPresent
command:
- sh
- /mq-server-start.sh
env:
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: ROCKETMQ_PROCESS_ROLE
value: broker
- name: NAMESRV_ADDR
value: rocketmq-nameserver-0.rocketmq-nameserver-headless.rocketmq.svc:9876
- name: ROCKETMQ_CONF_brokerId
value: "1"
- name: ROCKETMQ_CONF_brokerRole
value: "SLAVE"
- name: ROCKETMQ_JAVA_OPTIONS_HEAP
value: -Xms1G -Xmx1G
ports:
- containerPort: 10909
name: vip
protocol: TCP
- containerPort: 10911
name: main
protocol: TCP
- containerPort: 10912
name: ha
protocol: TCP
resources:
limits:
cpu: 2
memory: 1Gi
requests:
cpu: 50m
memory: 512Mi
readinessProbe:
failureThreshold: 6
initialDelaySeconds: 10
periodSeconds: 10
tcpSocket:
port: main
timeoutSeconds: 3
lifecycle:
preStop:
exec:
command: ["sh", "-c", "sleep 5; ./mqshutdown broker"]
volumeMounts:
- mountPath: /home/rocketmq/logs
name: broker-storage
subPath: rocketmq-broker/logs
- mountPath: /home/rocketmq/store
name: broker-storage
subPath: rocketmq-broker/store
- mountPath: /etc/rocketmq/broker-base.conf
name: broker-base-config
subPath: broker-base.conf
- mountPath: /mq-server-start.sh
name: mq-server-start-sh
subPath: mq-server-start.sh
dnsPolicy: ClusterFirst
terminationGracePeriodSeconds: 30
volumes:
- configMap:
items:
- key: broker-base.conf
path: broker-base.conf
name: rocketmq-server-config
name: broker-base-config
- configMap:
items:
- key: mq-server-start.sh
path: mq-server-start.sh
name: rocketmq-server-config
defaultMode: 0755
name: mq-server-start-sh
volumeClaimTemplates:
- metadata:
name: broker-storage
spec:
accessModes:
- ReadWriteOnce
storageClassName: local-path
resources:
requests:
storage: "100Mi"
- proxy
---
# Source: rocketmq-cluster/templates/proxy/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: "rocketmq-proxy"
namespace: rocketmq
labels:
helm.sh/chart: rocketmq-cluster-12.3.2
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
app.kubernetes.io/version: "5.3.0"
app.kubernetes.io/managed-by: Helm
spec:
minReadySeconds: 20
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
component: proxy
template:
metadata:
annotations:
checksum/config: 9323bc706d85f980c210e9823264a63548598b649c4935f9db6559d4fecbcc93
labels:
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
component: proxy
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 5
podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
component: proxy
topologyKey: kubernetes.io/hostname
securityContext:
fsGroup: 3000
runAsUser: 3000
containers:
- name: proxy
image: "apache/rocketmq:5.3.0"
imagePullPolicy: IfNotPresent
command:
- sh
- /mq-server-start.sh
env:
- name: NAMESRV_ADDR
value: rocketmq-nameserver-0.rocketmq-nameserver-headless.rocketmq.svc:9876
- name: ROCKETMQ_PROCESS_ROLE
value: proxy
- name: RMQ_PROXY_CONFIG_PATH
value: /etc/rocketmq/proxy.json
- name: ROCKETMQ_JAVA_OPTIONS_HEAP
value: -Xms1G -Xmx1G
ports:
- name: main
containerPort: 8080
protocol: TCP
- name: grpc
containerPort: 8081
protocol: TCP
resources:
limits:
cpu: 2
memory: 1Gi
requests:
cpu: 100m
memory: 512Mi
readinessProbe:
failureThreshold: 6
initialDelaySeconds: 10
periodSeconds: 10
tcpSocket:
port: main
timeoutSeconds: 3
lifecycle:
preStop:
exec:
command: ["sh", "-c", "sleep 5; ./mqshutdown proxy"]
volumeMounts:
- mountPath: /mq-server-start.sh
name: mq-server-start-sh
subPath: mq-server-start.sh
- mountPath: /etc/rocketmq/proxy.json
name: proxy-json
subPath: proxy.json
dnsPolicy: ClusterFirst
terminationGracePeriodSeconds: 15
volumes:
- configMap:
items:
- key: mq-server-start.sh
path: mq-server-start.sh
name: rocketmq-server-config
defaultMode: 0755
name: mq-server-start-sh
- configMap:
items:
- key: proxy.json
path: proxy.json
name: rocketmq-server-config
name: proxy-json
---
# Source: rocketmq-cluster/templates/proxy/service.yaml
apiVersion: v1
kind: Service
metadata:
name: rocketmq-proxy
labels:
helm.sh/chart: rocketmq-cluster-12.3.2
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
app.kubernetes.io/version: "5.3.0"
app.kubernetes.io/managed-by: Helm
component: proxy
spec:
ports:
- port: 8080
name: main
protocol: TCP
targetPort: 8080
- port: 8081
name: grpc
protocol: TCP
targetPort: 8081
selector:
app.kubernetes.io/name: rocketmq
app.kubernetes.io/instance: rocketmq
component: proxy
type: "ClusterIP"
- configmap
---
# Source: rocketmq-cluster/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: rocketmq-server-config
namespace: rocketmq
data:
broker-base.conf: |
deleteWhen = 04
fileReservedTime = 48
flushDiskType = ASYNC_FLUSH
waitTimeMillsInSendQueue = 1000
brokerClusterName = rocketmq-helm
controller-base.conf: |
controllerDLegerGroup = group1
enableElectUncleanMaster = false
notifyBrokerRoleChanged = true
controllerDLegerPeers = n0-rocketmq-controller-0.rocketmq-controller.rocketmq.svc:9878;n1-rocketmq-controller-1.rocketmq-controller.rocketmq.svc:9878;n2-rocketmq-controller-2.rocketmq-controller.rocketmq.svc:9878
controllerStorePath = /home/rocketmq/controller-data
proxy.json: |
{
"rocketMQClusterName": "rocketmq-helm"
}
mq-server-start.sh: |
java -version
if [ $? -ne 0 ]; then
echo "[ERROR] Missing java runtime"
exit 50
fi
if [ -z "${ROCKETMQ_HOME}" ]; then
echo "[ERROR] Missing env ROCKETMQ_HOME"
exit 50
fi
if [ -z "${ROCKETMQ_PROCESS_ROLE}" ]; then
echo "[ERROR] Missing env ROCKETMQ_PROCESS_ROLE"
exit 50
fi
export JAVA_HOME=$(dirname $(dirname $(readlink -f $(which java))))
export CLASSPATH=".:${ROCKETMQ_HOME}/conf:${ROCKETMQ_HOME}/lib/*:${CLASSPATH}"
JAVA_OPT="${JAVA_OPT} -server"
if [ -n "$ROCKETMQ_JAVA_OPTIONS_OVERRIDE" ]; then
JAVA_OPT="${JAVA_OPT} ${ROCKETMQ_JAVA_OPTIONS_OVERRIDE}"
else
JAVA_OPT="${JAVA_OPT} -XX:+UseG1GC"
JAVA_OPT="${JAVA_OPT} ${ROCKETMQ_JAVA_OPTIONS_EXT}"
JAVA_OPT="${JAVA_OPT} ${ROCKETMQ_JAVA_OPTIONS_HEAP}"
fi
JAVA_OPT="${JAVA_OPT} -cp ${CLASSPATH}"
export BROKER_CONF_FILE="$HOME/broker.conf"
export CONTROLLER_CONF_FILE="$HOME/controller.conf"
update_broker_conf() {
local key=$1
local value=$2
sed -i "/^${key} *=/d" ${BROKER_CONF_FILE}
echo "${key} = ${value}" >> ${BROKER_CONF_FILE}
}
init_broker_role() {
if [ "${ROCKETMQ_CONF_brokerRole}" = "SLAVE" ]; then
update_broker_conf "brokerRole" "SLAVE"
elif [ "${ROCKETMQ_CONF_brokerRole}" = "SYNC_MASTER" ]; then
update_broker_conf "brokerRole" "SYNC_MASTER"
else
update_broker_conf "brokerRole" "ASYNC_MASTER"
fi
if echo "${ROCKETMQ_CONF_brokerId}" | grep -E '^[0-9]+$'; then
update_broker_conf "brokerId" "${ROCKETMQ_CONF_brokerId}"
fi
}
init_broker_conf() {
rm -f ${BROKER_CONF_FILE}
cp /etc/rocketmq/broker-base.conf ${BROKER_CONF_FILE}
echo "" >> ${BROKER_CONF_FILE}
echo "# generated config" >> ${BROKER_CONF_FILE}
broker_name_seq=${HOSTNAME##*-}
if [ -n "$MY_POD_NAME" ]; then
broker_name_seq=${MY_POD_NAME##*-}
fi
update_broker_conf "brokerName" "broker-g${broker_name_seq}"
if [ "$enableControllerMode" != "true" ]; then
init_broker_role
fi
echo "[exec] cat ${BROKER_CONF_FILE}"
cat ${BROKER_CONF_FILE}
}
init_acl_conf() {
if [ -f /etc/rocketmq/acl/plain_acl.yml ]; then
rm -f "${ROCKETMQ_HOME}/conf/plain_acl.yml"
ln -sf "/etc/rocketmq/acl" "${ROCKETMQ_HOME}/conf/acl"
fi
}
init_controller_conf() {
rm -f ${CONTROLLER_CONF_FILE}
cp /etc/rocketmq/base-cm/controller-base.conf ${CONTROLLER_CONF_FILE}
controllerDLegerSelfId="n${HOSTNAME##*-}"
if [ -n "$MY_POD_NAME" ]; then
controllerDLegerSelfId="n${MY_POD_NAME##*-}"
fi
sed -i "/^controllerDLegerSelfId *=/d" ${CONTROLLER_CONF_FILE}
echo "controllerDLegerSelfId = ${controllerDLegerSelfId}" >> ${CONTROLLER_CONF_FILE}
cat ${CONTROLLER_CONF_FILE}
}
if [ "$ROCKETMQ_PROCESS_ROLE" = "broker" ]; then
init_broker_conf
init_acl_conf
set -x
java ${JAVA_OPT} org.apache.rocketmq.broker.BrokerStartup -c ${BROKER_CONF_FILE}
elif [ "$ROCKETMQ_PROCESS_ROLE" = "controller" ]; then
init_controller_conf
set -x
java ${JAVA_OPT} org.apache.rocketmq.controller.ControllerStartup -c ${CONTROLLER_CONF_FILE}
elif [ "$ROCKETMQ_PROCESS_ROLE" = "nameserver" ] || [ "$ROCKETMQ_PROCESS_ROLE" = "mqnamesrv" ]; then
set -x
if [ "$enableControllerInNamesrv" = "true" ]; then
init_controller_conf
java ${JAVA_OPT} org.apache.rocketmq.namesrv.NamesrvStartup -c ${CONTROLLER_CONF_FILE}
else
java ${JAVA_OPT} org.apache.rocketmq.namesrv.NamesrvStartup
fi
elif [ "$ROCKETMQ_PROCESS_ROLE" = "proxy" ]; then
set -x
if [ -f $RMQ_PROXY_CONFIG_PATH ]; then
java ${JAVA_OPT} org.apache.rocketmq.proxy.ProxyStartup -pc $RMQ_PROXY_CONFIG_PATH
else
java ${JAVA_OPT} org.apache.rocketmq.proxy.ProxyStartup
fi
else
echo "[ERROR] Missing env ROCKETMQ_PROCESS_ROLE"
exit 50
fi
踩坑
- 存儲(chǔ)權(quán)限問(wèn)題
配置完pv后啟動(dòng)一直報(bào)錯(cuò)。
查日志：
kubectl describe pod rocketmq-broker-master-0 -n rocketmq-demo
結(jié)果：
查應(yīng)用啟動(dòng)日志：
kubectl logs rocketmq-broker-master-0 -n rocketmq-demo
結(jié)果：
具體錯(cuò)誤信息：
03:30:58,822 |-ERROR in org.apache.rocketmq.logging.ch.qos.logback.core.rolling.RollingFileAppender[RocketmqAuthAuditAppender_inner] - Failed to create parent directories for [/home/rocketmq/logs/rocketmqlogs/auth_audit.log]
03:30:58,822 |-ERROR in org.apache.rocketmq.logging.ch.qos.logback.core.rolling.RollingFileAppender[RocketmqAuthAuditAppender_inner] - openFile(/home/rocketmq/logs/rocketmqlogs///auth_audit.log,true) call failed. java.io.FileNotFoundException: /home/rocketmq/logs/rocketmqlogs/auth_audit.log (No such file or directory)
at java.io.FileNotFoundException: /home/rocketmq/logs/rocketmqlogs/auth_audit.log (No such file or directory)
java.lang.NullPointerException
at org.apache.rocketmq.broker.schedule.ScheduleMessageService.configFilePath(ScheduleMessageService.java:272)
at org.apache.rocketmq.common.ConfigManager.persist(ConfigManager.java:83)
at org.apache.rocketmq.broker.BrokerController.shutdownBasicService(BrokerController.java:1478)
at org.apache.rocketmq.broker.BrokerController.shutdown(BrokerController.java:1565)
at org.apache.rocketmq.broker.BrokerStartup.createBrokerController(BrokerStartup.java:250)
at org.apache.rocketmq.broker.BrokerStartup.main(BrokerStartup.java:52)
網(wǎng)上查了下是掛載的本地目錄，pod沒(méi)有權(quán)限讀寫(xiě)。
解決的方式：
1、移出root目錄
由于是root用戶(hù)賬號(hào)，k8s啟動(dòng)用的kubectl賬號(hào)，把掛載的目錄移到了/tmp，修改上文PV文件。
2、提前創(chuàng)建PV目錄
在/tmp目錄下創(chuàng)建/tmp/storage，不然啟動(dòng)會(huì)報(bào)PVC沒(méi)有該目錄
3、chmod開(kāi)啟目錄及子目錄的讀寫(xiě)權(quán)限，需要帶上-R遞歸修改所有子目錄
chmod -R 777 storage
- 主從副本問(wèn)題
修改完文件權(quán)限啟動(dòng)后又報(bào)以下錯(cuò)誤：
而且Broker一主一從，一個(gè)正常啟動(dòng)，另一個(gè)報(bào)這個(gè)錯(cuò)誤。網(wǎng)上查了下，正常會(huì)出現(xiàn)在同一個(gè)機(jī)器部署了兩個(gè)Broker的情況下。
但我們的環(huán)境是K8S集群，節(jié)點(diǎn)之間理應(yīng)是隔離的，所以猜想是storage掛載了同一個(gè)目錄的問(wèn)題，修改PV，兩個(gè)PV掛載的目錄不同，改為storageSlave。再次啟動(dòng)后成功。
- 命名空間問(wèn)題
轉(zhuǎn)移到開(kāi)發(fā)環(huán)境后報(bào)，啟動(dòng)Broker和Namesrver正常，但proxy啟動(dòng)不了，報(bào)：
org.apache.rocketmq.proxy.common.ProxyException: create system broadcast topic DefaultHeartBeatSyncerTopic failed on cluster rocketmq-helm
在本地環(huán)境轉(zhuǎn)完yaml啟動(dòng)時(shí)沒(méi)報(bào)過(guò)。
網(wǎng)上查了下，如果broker沒(méi)正確配置nameserver，也會(huì)報(bào)這個(gè)錯(cuò)誤。懷疑是環(huán)境變了后，某些配置需要根據(jù)環(huán)境修改。把目錄下的配置都仔細(xì)研究了下，尤其涉及broker和proxy的nameserver地址的。
還有個(gè)地方有差異，由于開(kāi)發(fā)環(huán)境多人共用，有許多的應(yīng)用在跑，而導(dǎo)出的yaml文件會(huì)默認(rèn)在K8S的default namespace啟動(dòng)pod，容易造成混亂和不好管理。所以嘗試在yaml文件中加入了namespace:rocketmq。
最后排查確實(shí)由于這個(gè)導(dǎo)致，在proxy和broker的配置文件中，還有這句讀取nameserver地址的語(yǔ)句：
需要將其中的：
value: rocketmq-nameserver-0.rocketmq-nameserver-headless.default.svc:9876
改為：
value: rocketmq-nameserver-0.rocketmq-nameserver-headless.rocketmq.svc:9876
這個(gè)環(huán)境變量提供了 RocketMQ NameServer 的地址和端口。rocketmq-nameserver-0.rocketmq-nameserver-headless.default.svc 是 NameServer Pod 的 DNS 名稱(chēng)，9876 是 NameServer 服務(wù)的端口。這個(gè)地址用于客戶(hù)端或 Broker 連接到 NameServer，以便進(jìn)行服務(wù)發(fā)現(xiàn)和元數(shù)據(jù)同步。
其中，rocketmq-nameserver-0是當(dāng)前nameserver的name，rocketmq-nameserver-headless對(duì)應(yīng)headless service的name，default對(duì)應(yīng)namespace。所以部署新的K8S命名空間后，需要也把這里的default改為rocketmq的namespace，否則就會(huì)報(bào)找不到無(wú)法創(chuàng)建topit的錯(cuò)誤。不過(guò)，這里挺奇怪的，broker能正常啟動(dòng)，只有啟動(dòng)proxy的時(shí)候才會(huì)報(bào)這個(gè)錯(cuò)誤，估計(jì)Rocketmq5新版本做了什么修改。
看了別人debug啟動(dòng)源碼，指出是這里的問(wèn)題：
在BrokerStartup.java閱讀發(fā)現(xiàn)這個(gè)是namesrv的地址，如果不添加的話(huà)，會(huì)導(dǎo)致即使你啟動(dòng)了broker，但其實(shí)并不會(huì)在namesrv上有任何注冊(cè)信息。
如果不配置會(huì)發(fā)生什么呢，主要體現(xiàn)在proxy啟動(dòng)的時(shí)候，就一定會(huì)報(bào)錯(cuò)
create system broadcast topic DefaultHeartBeatSyncerTopic failed on cluster DefaultCluster
總結(jié)
以上為個(gè)人經(jīng)驗(yàn)，希望能給大家一個(gè)參考，也希望大家多多支持腳本之家。
相關(guān)文章
這篇文章主要介紹了K8S?中?kubectl?命令,本文給大家介紹的非常詳細(xì)，對(duì)大家的學(xué)習(xí)或工作具有一定的參考借鑒價(jià)值，需要的朋友可以參考下2022-07-07
Rainbond功能架構(gòu)及應(yīng)用管理官方文檔介紹
這篇文章主要為大家介紹了Rainbond功能機(jī)構(gòu)及使用官方文檔，有需要的朋友可以借鑒參考下，希望能夠有所幫助，祝大家多多進(jìn)步，早日升職加薪2022-04-04
如何在 K8S 中使用 Values 文件定制不同環(huán)境下的應(yīng)用配置
Kubernetes是一個(gè)開(kāi)源的容器編排平臺(tái),它可以自動(dòng)化容器的部署、擴(kuò)展和管理,在 K8s 中,應(yīng)用程序通常以容器的形式運(yùn)行,這些容器被組織在不同的資源對(duì)象中,這篇文章主要介紹了如何在 K8S 中使用 Values 文件定制不同環(huán)境下的應(yīng)用配置,需要的朋友可以參考下2025-03-03
K8S-ConfigMap實(shí)現(xiàn)應(yīng)用和配置分離詳解
這篇文章主要為大家介紹了K8S-ConfigMap實(shí)現(xiàn)應(yīng)用和配置分離詳解，有需要的朋友可以借鑒參考下，希望能夠有所幫助，祝大家多多進(jìn)步，早日升職加薪2023-04-04
本文主要介紹了podman容器工具的具體使用，文中通過(guò)示例代碼介紹的非常詳細(xì)，具有一定的參考價(jià)值，感興趣的小伙伴們可以參考一下2022-01-01
二進(jìn)制方式安裝?Kubernetes1.18.3版本實(shí)現(xiàn)腳本
這篇文章主要為大家介紹了二進(jìn)制方式安裝Kubernetes1.18.3版本實(shí)現(xiàn)腳本，希望能夠有所幫助，祝大家多多進(jìn)步，早日升職加薪2022-03-03
Kubernetes應(yīng)用服務(wù)質(zhì)量管理詳解
這篇文章主要為大家介紹了Kubernetes應(yīng)用服務(wù)質(zhì)量管理詳解，有需要的朋友可以借鑒參考下，希望能夠有所幫助，祝大家多多進(jìn)步，早日升職加薪2022-11-11
這篇文章主要給大家介紹了關(guān)于K8S中五種控制器及使用的相關(guān)資料,控制器 又稱(chēng)之為工作負(fù)載,本文通過(guò)圖文以及實(shí)例代碼介紹的非常詳細(xì),需要的朋友可以參考下2021-12-12
K8S?實(shí)用工具之合并多個(gè)kubeconfig實(shí)現(xiàn)詳解
這篇文章主要為大家介紹了K8S?實(shí)用工具之合并多個(gè)kubeconfig實(shí)現(xiàn)詳解，有需要的朋友可以借鑒參考下，希望能夠有所幫助，祝大家多多進(jìn)步，早日升職加薪2023-03-03
Kubernetes中使用臨時(shí)容器進(jìn)行故障排查的方法
在使用Kubernetes時(shí),用戶(hù)常常會(huì)遇到一些錯(cuò)誤和迷惑,下面這篇文章主要給大家介紹了關(guān)于Kubernetes中使用臨時(shí)容器進(jìn)行故障排查的相關(guān)資料,需要的朋友可以參考下2022-03-03