Flink SQL FileSystem Connector 分区提交与自定义小文件合并策略

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Flink SQL 的 FileSystem Connector 为了与 Flink-Hive 集成的大环境适配，做了很多改进，而其中最为明显的就是分区提交（partition commit）机制。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文先通过源码简单过一下分区提交机制的两个要素——即触发（trigger）和策略（policy）的实现，然后用合并小文件的实例说一下自定义分区提交策略的方法。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"PartitionCommitTrigger"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"path\n└── datetime=2019-08-25\n └── hour=11\n ├── part-0.parquet\n ├── part-1.parquet\n └── hour=12\n ├── part-0.parquet\n└── datetime=2019-08-26\n └── hour=6\n ├── part-0.parquet"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那么，已经写入的分区数据何时才能对下游可见呢？这就涉及到如何触发分区提交的问题。根据官方文档，触发参数有以下两个："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"sink.partition-commit.trigger"},{"type":"text","text":"：可选 process-time（根据处理时间触发）和 partition-time（根据从事件时间中提取的分区时间触发）。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"sink.partition-commit.delay"},{"type":"text","text":"：分区提交的时延。如果 trigger 是 process-time，则以分区创建时的系统时间戳为准，经过此时延后提交；如果 trigger 是 partition-time，则以分区创建时本身携带的事件时间戳为准，当水印时间戳经过此时延后提交。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可见，process-time trigger 无法应对处理过程中出现的抖动，一旦数据迟到或者程序失败重启，数据就不能按照事件时间被归入正确的分区了。所以在实际应用中，我们几乎总是选用 partition-time trigger，并自己生成水印。当然我们也需要通过 partition.time-extractor.*一系列参数来指定抽取分区时间的规则（PartitionTimeExtractor），官方文档说得很清楚，不再赘述。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在源码中，PartitionCommitTrigger 的类图如下。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e8/e8628dd73fb5cfba75c08cc46016cec4.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面以分区时间触发的 PartitionTimeCommitTrigger 为例，简单看看它的思路。直接上该类的完整代码。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"public class PartitionTimeCommitTigger implements PartitionCommitTrigger {\n private static final ListStateDescriptor> PENDING_PARTITIONS_STATE_DESC =\n new ListStateDescriptor<>(\n \"pending-partitions\",\n new ListSerializer<>(StringSerializer.INSTANCE));\n\n private static final ListStateDescriptor