kafka消費(fèi)不到數(shù)據(jù)的排查過程
kafka消費(fèi)不到數(shù)據(jù)的排查
集群上新安裝并啟動(dòng)了3個(gè)kafka Broker，代碼打包上傳至集群，運(yùn)行后發(fā)現(xiàn)一直消費(fèi)不到數(shù)據(jù)，
本地idea中debug后發(fā)現(xiàn)，程序一直阻塞在如下程序中，陷入了死循環(huán)。
/**
* Block until the coordinator for this group is known and is ready to receive requests.
* 等待直到我們和服務(wù)端的GroupCoordinator取得連接
*/
public void ensureCoordinatorReady() {
while (coordinatorUnknown()) {//無法獲取GroupCoordinator
RequestFuture<Void> future = sendGroupCoordinatorRequest();//發(fā)送請求
client.poll(future);//同步等待異步調(diào)用的結(jié)果
if (future.failed()) {
if (future.isRetriable())
client.awaitMetadataUpdate();
else
throw future.exception();
} else if (coordinator != null && client.connectionFailed(coordinator)) {
// we found the coordinator, but the connection has failed, so mark
// it dead and backoff before retrying discovery
coordinatorDead();
time.sleep(retryBackoffMs);//等待一段時(shí)間，然后重試
}
}
}
流程大概說就是
- consumer會(huì)從集群中選取一個(gè)broker作為coordinator
- 然后group中的consumer會(huì)向coordinator發(fā)請求申請成為consumergroup中的leader
- 最后有1個(gè)consumer會(huì)成為consumerLeader ,其他consumer成為follower
- consumerLeader做分區(qū)分配任務(wù),同步給coordinator
- consumerFollower從coordinator同步分區(qū)分配數(shù)據(jù)
問題出現(xiàn)在第一步，意思就是說Consumer和服務(wù)端的GroupCoordinator無法取得連接，所以程序一直在等待狀態(tài)。
看了下__consumer_offsets 這個(gè)topic情況,50個(gè)分區(qū)全在broker id為152的broker上
bin/kafka-topics.sh --describe --zookeeper localhost:2182 --topic __consumer_offsets
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:1 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
Topic: __consumer_offsets Partition: 0 Leader: 152 Replicas: 152 Isr:152
Topic: __consumer_offsets Partition: 1 Leader: 152 Replicas: 152 Isr:152
Topic: __consumer_offsets Partition: 2 Leader: 152 Replicas: 152 Isr:152
Topic: __consumer_offsets Partition: 3 Leader: 152
......
但是集群上并沒有broker id為152的節(jié)點(diǎn)，想到該集群kafka節(jié)點(diǎn)曾經(jīng)添加刪除過節(jié)點(diǎn)，初步斷定152是之前的kafka節(jié)點(diǎn)，后來該節(jié)點(diǎn)去掉后又加入新的節(jié)點(diǎn)但是zookeeper中的數(shù)據(jù)并沒有更新。
所以就關(guān)閉broker，進(jìn)入zookeeper客戶端，將brokers節(jié)點(diǎn)下的topics節(jié)點(diǎn)下的__consumer_offsets刪除，然后重啟broker，注意，此時(shí)zookeeper上__consumer_offsets還并沒有生成,要開啟消費(fèi)者之后才會(huì)生成.
然后再觀察__consumer_offsets,分區(qū)已經(jīng)均勻分布在三個(gè)broker上面了
bin/kafka-topics.sh --zookeeper localhost:2182 --describe --topic __consumer_offsets
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:3 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
Topic: __consumer_offsets Partition: 0 Leader: 420 Replicas: 420,421,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 1 Leader: 421 Replicas: 421,422,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 2 Leader: 422 Replicas: 422,420,421 Isr: 422,420,421
Topic: __consumer_offsets Partition: 3 Leader: 420 Replicas: 420,422,421 Isr: 420,422,421
Topic: __consumer_offsets Partition: 4 Leader: 421 Replicas: 421,420,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 5 Leader: 422 Replicas: 422,421,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 6 Leader: 420 Replicas: 420,421,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 7 Leader: 421 Replicas: 421,422,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 8 Leader: 422 Replicas: 422,420,421 Isr: 422,420,421
Topic: __consumer_offsets Partition: 9 Leader: 420 Replicas: 420,422,421 Isr: 420,422,421
Topic: __consumer_offsets Partition: 10 Leader: 421 Replicas: 421,420,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 11 Leader: 422 Replicas: 422,421,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 12 Leader: 420 Replicas: 420,421,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 13 Leader: 421 Replicas: 421,422,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 14 Leader: 422 Replicas: 422,420,421 Isr: 422,420,421
Topic: __consumer_offsets Partition: 15 Leader: 420 Replicas: 420,422,421 Isr: 420,422,421
Topic: __consumer_offsets Partition: 16 Leader: 421 Replicas: 421,420,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 17 Leader: 422 Replicas: 422,421,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 18 Leader: 420 Replicas: 420,421,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 19 Leader: 421 Replicas: 421,422,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 20 Leader: 422 Replicas: 422,420,421 Isr: 422,420,421
Topic: __consumer_offsets Partition: 21 Leader: 420 Replicas: 420,422,421 Isr: 420,422,421
Topic: __consumer_offsets Partition: 22 Leader: 421 Replicas: 421,420,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 23 Leader: 422 Replicas: 422,421,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 24 Leader: 420 Replicas: 420,421,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 25 Leader: 421 Replicas: 421,422,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 26 Leader: 422 Replicas: 422,420,421 Isr: 422,420,421
Topic: __consumer_offsets Partition: 27 Leader: 420 Replicas: 420,422,421 Isr: 420,422,421
Topic: __consumer_offsets Partition: 28 Leader: 421 Replicas: 421,420,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 29 Leader: 422 Replicas: 422,421,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 30 Leader: 420 Replicas: 420,421,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 31 Leader: 421 Replicas: 421,422,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 32 Leader: 422 Replicas: 422,420,421 Isr: 422,420,421
Topic: __consumer_offsets Partition: 33 Leader: 420 Replicas: 420,422,421 Isr: 420,422,421
Topic: __consumer_offsets Partition: 34 Leader: 421 Replicas: 421,420,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 35 Leader: 422 Replicas: 422,421,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 36 Leader: 420 Replicas: 420,421,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 37 Leader: 421 Replicas: 421,422,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 38 Leader: 422 Replicas: 422,420,421 Isr: 422,420,421
Topic: __consumer_offsets Partition: 39 Leader: 420 Replicas: 420,422,421 Isr: 420,422,421
Topic: __consumer_offsets Partition: 40 Leader: 421 Replicas: 421,420,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 41 Leader: 422 Replicas: 422,421,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 42 Leader: 420 Replicas: 420,421,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 43 Leader: 421 Replicas: 421,422,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 44 Leader: 422 Replicas: 422,420,421 Isr: 422,420,421
Topic: __consumer_offsets Partition: 45 Leader: 420 Replicas: 420,422,421 Isr: 420,422,421
Topic: __consumer_offsets Partition: 46 Leader: 421 Replicas: 421,420,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 47 Leader: 422 Replicas: 422,421,420 Isr: 422,420,421
Topic: __consumer_offsets Partition: 48 Leader: 420 Replicas: 420,421,422 Isr: 420,422,421
Topic: __consumer_offsets Partition: 49 Leader: 421 Replicas: 421,422,420 Isr: 422,420,421
這個(gè)時(shí)候重啟程序，發(fā)現(xiàn)已經(jīng)可以正常消費(fèi)了，問題解決。
參考資料：
總結(jié)
以上為個(gè)人經(jīng)驗(yàn)，希望能給大家一個(gè)參考，也希望大家多多支持腳本之家。
相關(guān)文章
在開發(fā)中,有時(shí)需要獲取Java類的屬性注釋,尤其是當(dāng)JPA生成的表缺少注釋時(shí),可以通過jdk自帶的tools.jar工具包來實(shí)現(xiàn),方法類似于生成javadoc文檔,需要在pom.xml文件中導(dǎo)入tools.jar的依賴,該jar文件一般位于JAVA_HOME/lib目錄下2024-09-09
java.io.UnsupportedEncodingException異常的正確解決方法(親測有效！)
這篇文章主要給大家介紹了關(guān)于java.io.UnsupportedEncodingException異常的正確解決方法,文中介紹的辦法親測有效,java.io.UnsupportedEncodingException是Java編程語言中的一個(gè)異常類,表示指定的字符集不被支持,需要的朋友可以參考下2024-02-02
如何解決Gradle、Maven項(xiàng)目build后沒有mybatis的mapper.xml文件的問題
這篇文章主要介紹了如何解決Gradle、Maven項(xiàng)目build后沒有mybatis的mapper.xml文件的問題,具有很好的參考價(jià)值,希望對大家有所幫助,如有錯(cuò)誤或未考慮完全的地方,望不吝賜教2024-01-01
Java中Cron表達(dá)式的生成解析及計(jì)算的工具類完整代碼
這篇文章主要給大家介紹了關(guān)于Java中Cron表達(dá)式的生成解析及計(jì)算工具類的相關(guān)資料,Cron表達(dá)式是一個(gè)字符串,字符串空格分割,每一個(gè)域代表一個(gè)含義,一個(gè)cron表達(dá)式有至少6個(gè),需要的朋友可以參考下2023-12-12
除了Java本身的字符串處理方式外，我們還可以使用Apache Common Langs里的StringUtils來簡化String的操作。但以上兩種方式對于我們?nèi)粘＞幊讨凶钊菀着龅降淖址幚韥碚f，仍然顯得有些不足。所以這篇文章給大家介紹Java8中字符串處理庫strman-java的使用。2016-09-09
java多線程編程之向線程傳遞數(shù)據(jù)的三種方法
在多線程的異步開發(fā)模式下，數(shù)據(jù)的傳遞和返回和同步開發(fā)模式有很大的區(qū)別。由于線程的運(yùn)行和結(jié)束是不可預(yù)料的，因此，在傳遞和返回?cái)?shù)據(jù)時(shí)就無法象函數(shù)一樣通過函數(shù)參數(shù)和return語句來返回?cái)?shù)據(jù)2014-01-01
簡單幾步實(shí)現(xiàn)將Spring security4.x升級(jí)到5.x
這篇文章主要介紹了簡單幾步實(shí)現(xiàn)將Spring security4.x升級(jí)到5.x方式,具有很好的參考價(jià)值,希望對大家有所幫助,如有錯(cuò)誤或未考慮完全的地方,望不吝賜教2024-08-08
Java8中Stream使用的一個(gè)注意事項(xiàng)
最近在工作中發(fā)現(xiàn)了對于集合操作轉(zhuǎn)換的神器，java8新特性 stream，但在使用中遇到了一個(gè)非常重要的注意點(diǎn)，所以這篇文章主要給大家介紹了關(guān)于Java8中Stream使用過程中的一個(gè)注意事項(xiàng)，需要的朋友可以參考借鑒，下面隨著小編來一起學(xué)習(xí)學(xué)習(xí)吧。2017-11-11
這篇文章主要介紹了java隨機(jī)事件分發(fā)器示例,需要的朋友可以參考下,功能需求和代碼實(shí)現(xiàn)在下面2014-03-03
Netty分布式pipeline管道傳播outBound事件源碼解析
這篇文章主要介紹了Netty分布式pipeline管道傳播outBound事件源碼解析，有需要的朋友可以借鑒參考下，希望能夠有所幫助，祝大家多多進(jìn)步，早日升職加薪2022-03-03