druid.io中kafka-index-service移交多次失敗的問題排查與處理

原創 梦想成真那天 2020-02-22 14:59

+問題日誌如下:

2020-02-02T23:41:03,855 INFO [publish-0] io.druid.indexing.kafka.KafkaIndexTask - Persisting Sequences Metadata [[]]
2020-02-02T23:41:03,856 INFO [publish-0] io.druid.segment.realtime.appenderator.StreamAppenderatorDriver - Register handoff of segments: [[PROD_CTI_P1D_P1D_2020-02-03T00:00:00.000Z_2020-02-04T00:00:00.000Z_2020-02-02T16:10:43.841Z_33]]
2020-02-02T23:41:03,858 INFO [publish-0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Adding SegmentHandoffCallback for dataSource[PROD_CTI_P1D_P1D] Segment[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]
2020-02-02T23:42:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-02T23:42:00,214 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-02T23:43:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-02T23:43:00,211 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-02T23:44:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-02T23:44:00,210 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-02T23:45:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-02T23:45:00,210 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-02T23:46:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-02T23:46:00,210 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-02T23:47:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-02T23:47:00,210 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-02T23:48:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-02T23:48:00,210 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-02T23:49:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-02T23:49:00,211 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-02T23:50:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-02T23:50:00,211 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-02T23:51:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-02T23:51:00,211 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-02T23:52:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-02T23:52:00,210 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-02T23:53:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-02T23:53:00,210 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-02T23:54:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-02T23:54:00,210 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-02T23:55:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-02T23:55:00,210 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-02T23:56:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-02T23:56:00,211 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-02T23:57:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-02T23:57:00,212 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-02T23:58:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-02T23:58:00,210 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-02T23:59:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-02T23:59:00,210 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-03T00:00:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-03T00:00:00,210 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-03T00:01:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-03T00:01:00,210 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-03T00:02:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-03T00:02:00,209 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-03T00:03:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-03T00:03:00,210 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-03T00:04:00,212 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-03T00:04:00,232 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-03T00:05:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-03T00:05:00,209 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-03T00:06:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-03T00:06:00,210 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-03T00:07:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-03T00:07:00,210 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-03T00:08:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-03T00:08:00,210 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-03T00:09:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-03T00:09:00,213 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-03T00:10:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-03T00:10:00,210 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]
2020-02-03T00:11:00,205 INFO [coordinator_handoff_scheduled_0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://emr-header-1.cluster-66264:18081
2020-02-03T00:11:00,210 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2020-02-03T00:00:00.000Z/2020-02-04T00:00:00.000Z, version='2020-02-02T16:10:43.841Z', partitionNumber=33}]]

該類問題主要是數據已經落地到deepstorage，但是coredinator無法加載到這個segments，一般這類的問題都是設置的ruls存在問題，我這個問題的發生時設置的規則dropForever，加載8個小時前的數據，最近8個小時的數據由於設置的規則不允許加載，所以造成了cordinator節點無法加載到最難8小時segment，並生成緩存文件的句柄，因此任務一直時間，無法加載到系統中。然後對於這類的問題，可以考慮是否是規則設置的有問題。

參考：
https://blog.csdn.net/bigtree_3721/article/details/80153365

夢想成真那天
發佈了118 篇原創文章 · 獲贊 20 · 訪問量 25萬+
私信 關注
發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章

druid.io數據遷移之insert-segment-to-db Tool

在druid.io的services目錄下,有比較多的druid.io的使用小工具,本結要說的就是這個insert-segment-to-db Tool工具,這個工具可以根據用戶的druid.io的segments數據來對元數據進

梦想成真那天 2020-06-27 12:17:59

druid.io之merge task

梦想成真那天 2019-02-27 21:23:51

druid.io數據遷移之insert-segment-to-db Tool

在druid.io的services目錄下,有比較多的druid.io的使用小工具,本結要說的就是這個insert-segment-to-db Tool工具,這個工具可以根據用戶的druid.io的segments數據來對元數據進

梦想成真那天 2020-06-27 12:17:59

io.druid.java.util.common.ISE: Could not allocate segment for row with timestamp

今天使用阿里雲druid.io服務,發現kafka-index-service這個任務都以失敗結束,查詢錯誤日誌,錯誤如下 io.druid.java.util.common.ISE: Could not allocate se

梦想成真那天 2020-06-27 12:17:59

druid.io數據重建之insert-segment-to-db

java -Ddruid.metadata.storage.type=mysql -Ddruid.metadata.storage.connector.connectURI=jdbc:mysql://emr-header-1:3

梦想成真那天 2020-06-27 12:17:59

Druid(Druid.io)架構分析

Druid架構分析 一、基本概念 Datasource：Druid中的Datasource可以理解爲Mysql中的表，但Druid對“表”中的”列”進行了分類，其包含下面三個重要的概念：  1. 時間列(Timestamp):每行數據的時

凌晨1点21分 2020-06-20 15:26:25

druid.io架構的個人理解學習 part1 part2 翻譯

master-dragon 2019-09-16 02:56:38

druid.io架構的個人理解學習

master-dragon 2019-09-16 02:56:38

druid.io值kafka-index-service任務失敗異常排查

梦想成真那天 2019-07-11 00:58:19

druid.oi之Group BYJSON查詢語法

梦想成真那天 2019-03-23 00:17:50

druid.io出現的問題及解決方法

youyoufusheng 2019-03-10 03:01:07

kafka-index-service實現一個topic多個不同查詢粒度的kafka-index-service任務

梦想成真那天 2018-12-12 20:53:57

第一章durid.io簡介

fish_cool 2018-09-05 03:15:42

第四章：druid.io的功能

fish_cool 2018-09-05 03:15:42

第七章：druid.io實踐分享之realtime+kafka 一

fish_cool 2018-09-05 03:15:42