一、checkpoint失敗
原因：我把insert into這個語句給註釋掉了
其他的原因大概率是flinksql不夠嚴謹
二、運行中的報錯
java.io.IOException: Failed to deserialize consumer record due to
at org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter.emitRecord(KafkaRecordEmitter.java:54)
at org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter.emitRecord(KafkaRecordEmitter.java:32)
at org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:143)
at org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:354)
at org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:496)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Failed to deserialize consumer record ConsumerRecord(topic = topic_test_01, partition = 2, leaderEpoch = 0, offset = 808016, CreateTime = 1682242261803, serialized key size = -1, serialized value size = 278, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = [B@25f6134a).
at org.apache.flink.connector.kafka.source.reader.deserializer.KafkaDeserializationSchemaWrapper.deserialize(KafkaDeserializationSchemaWrapper.java:57)
at org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter.emitRecord(KafkaRecordEmitter.java:51)
... 14 more
Caused by: org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException: Could not forward element to next operator
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:99)
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57)
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
at org.apache.flink.streaming.runtime.tasks.SourceOperatorStreamTask$AsyncDataOutputToOutput.emitRecord(SourceOperatorStreamTask.java:196)
at org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:110)
at org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter$SourceOutputWrapper.collect(KafkaRecordEmitter.java:65)
at org.apache.flink.api.common.serialization.DeserializationSchema.deserialize(DeserializationSchema.java:84)
at org.apache.flink.streaming.connectors.kafka.table.DynamicKafkaDeserializationSchema.deserialize(DynamicKafkaDeserializationSchema.java:113)
at org.apache.flink.connector.kafka.source.reader.deserializer.KafkaDeserializationSchemaWrapper.deserialize(KafkaDeserializationSchemaWrapper.java:54)
... 15 more
Caused by: org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException: Could not forward element to next operator
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:99)
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57)
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56)
at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:29)
at StreamExecCalc$41.processElement_split3(Unknown Source)
at StreamExecCalc$41.processElement(Unknown Source)
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:82)
... 23 more
Caused by: org.apache.flink.kafka.shaded.org.apache.kafka.common.errors.TimeoutException: Topic test_tpoic_02 not present in metadata after 60000 ms.
這個錯誤消息表明Flink在反序列化來自Kafka主題“dwd_crm_mianxin_card_use_detail_rt”的消費者記錄時遇到了麻煩。
這個錯誤可能有幾個原因，但一個常見的原因是Kafka主題中的數據不是預期的格式。Flink使用反序列化器將Kafka主題中的二進制數據轉換爲Java對象，如果數據不是預期的格式，反序列化器將失敗。
要解決此問題，您可以嘗試以下步驟:
檢查Kafka主題中的數據以確保它是預期的格式。你可以使用Kafka消費者工具從主題中讀取數據並檢查它。
檢查Flink作業中Kafka消費者的配置，以確保它對主題中的數據使用了正確的反序列化器。你可以在Flink Kafka消費者配置中指定反序列化類。
檢查Flink和Kafka庫的版本兼容性。確保你正在使用的Flink和Kafka庫的版本相互兼容。
請檢查Flink作業的日誌，瞭解更詳細的錯誤信息。您提供的錯誤消息不是非常具體，因此日誌中可能有更多信息可以幫助您診斷問題。
三、向doris導入數據遇到的問題
1、table Already Exists and load job finished,change you label prefix or restore from lates savepoint
原因：這個是在任務重新啓動的時候，沒有修改參數：sink.label-prefix 的值，每次重啓任務向doris導數都得改其value。 例如：'sink.label-prefix'='label_ads_cdp_mianxin_card_use_detail_rt_pt_d_2023042716'。如果不改就會報上面的錯誤。
2、任務正常運行，沒有報錯，但是doris的目標表沒有數據
主要是把配置項修改得符合數據改：'sink.properties.strip_outer_array'='false',
加：'sink.properties.read_json_by_line'='true',
上圖錯誤主要是sink.properties.strip_outer_array=false 沒有設置或者說沒有生效
3、標準的導入doris的建表語句
create table if not exists ads_rt.ads_cdp_mianxin_card_use_detail_rt(
id string,
dt string,
date_happened string,
mbr_id string,
card_id string,
business_type string,
card_status string,
event_type string,
ts string,
balance decimal(22,2),
change_amount decimal(22,2)
) WITH (
'connector'='doris',
'fenodes'='10.00.00.01:8045',
'table.identifier'='doris_db.doris_sink_table',
'username'='username',
'password'='000000',
'sink.properties.format'='json',
'sink.properties.strip_outer_array'='false',
'sink.enable-delete'='true',
'sink.properties.read_json_by_line'='true',
'sink.label-prefix'='label_doris_sink_table_2023042716'
);