【Flume】HDFSSink配置参数说明

Name	Default	Description
channel	–
type	–	The component type name, needs to be hdfs
hdfs.path	–	HDFS directory path (eg hdfs://namenode/flume/webdata/)
hdfs.filePrefix	FlumeData	HDSFSink产生文件的前缀名称，如果定义为 MyFile，则生成文件将会是/hdfspath/MyFile.后缀
hdfs.fileSuffix	–	定义产生文件的后缀。例如要生成文本类型的.txt作为后缀
hdfs.inUsePrefix	–	写入HDFS过程中，会生成一个临时文件，inUsePrefix则定义了临时文件的前缀
hdfs.inUseSuffix	.tmp	定义写入HDFS过程中，文件的后缀
hdfs.rollInterval	30	Number of seconds to wait before rolling current file (0 = never roll based on time interval) 按时间生成HDFS文件
hdfs.rollSize	1024	File size to trigger roll, in bytes (0: never roll based on file size) 按文件大小触发roll，生成新的HDFS文件
hdfs.rollCount	10	Number of events written to file before it rolled (0 = never roll based on number of events) 按写入的event的个数触发roll，生成新的HDFS文件
hdfs.idleTimeout	0	Timeout after which inactive files get closed (0 = disable automatic closing of idle files) 当达到idleTimeout时，关闭空闲的文件（默认为0，不自动关闭空闲的文件）
hdfs.batchSize	100	number of events written to file before it is flushed to HDFS HDFSEventSink以batchSize的event个数作为一个事务处理，统一flush到HDFS中，提交事务
hdfs.codeC	–	Compression codec. one of following : gzip, bzip2, lzo, lzop, snappy 设置具体的压缩格式，在配置fileType为CompressedStream时，需要设置CodeC
hdfs.fileType	SequenceFile	File format: currently SequenceFile, DataStream or CompressedStream (1)DataStream will not compress output file and please don’t set codeC (2)CompressedStream requires set hdfs.codeC with an available codeC 在生成HDFSWriter的具体实现时，通过该参数制定HDFSWriter的实现类。 三个格式分别对应HDFSSequenceFile，HDFSDataStream，HDFSCompressedDataStream
hdfs.maxOpenFiles	5000	Allow only this number of open files. If this number is exceeded, the oldest file is closed. HDFSSink中维护了与HDFS创建的文件的连接（每个文件对应一个BucketWriter）， 如果超过该值，HDFSSink将会关闭创建最久的BucketWriter
hdfs.minBlockReplicas	–	Specify minimum number of replicas per HDFS block. If not specified, it comes from the default Hadoop config in the classpath. 设置HDFS块的副本数，如果没有特殊设置，默认采用Hadoop的配置
hdfs.writeFormat	Writable	Format for sequence file records. One of Text or Writable. Set to Text before creating data files with Flume, otherwise those files cannot be read by either Apache Impala (incubating) or Apache Hive.
hdfs.callTimeout	10000	Number of milliseconds allowed for HDFS operations, such as open, write, flush, close. This number should be increased if many HDFS timeout operations are occurring.
hdfs.threadsPoolSize	10	Number of threads per HDFS sink for HDFS IO ops (open, write, etc.) 处理HDFS相关操作的线程数
hdfs.rollTimerPoolSize	1	Number of threads per HDFS sink for scheduling timed file rolling 对HDFS文件进行roll操作的线程数
hdfs.kerberosPrincipal	–	Kerberos user principal for accessing secure HDFS
hdfs.kerberosKeytab	–	Kerberos keytab for accessing secure HDFS
hdfs.proxyUser
hdfs.round	false	Should the timestamp be rounded down (if true, affects all time based escape sequences except %t)
hdfs.roundValue	1	Rounded down to the highest multiple of this (in the unit configured using hdfs.roundUnit), less than current time.
hdfs.roundUnit	second	The unit of the round down value - second, minute or hour.
hdfs.timeZone	Local Time	Name of the timezone that should be used for resolving the directory path, e.g. America/Los_Angeles.
hdfs.useLocalTimeStamp	false	Use the local time (instead of the timestamp from the event header) while replacing the escape sequences.
hdfs.closeTries	0	Number of times the sink must try renaming a file, after initiating a close attempt. If set to 1, this sink will not re-try a failed rename (due to, for example, NameNode or DataNode failure), and may leave the file in an open state with a .tmp extension. If set to 0, the sink will try to rename the file until the file is eventually renamed (there is no limit on the number of times it would try). The file may still remain open if the close call fails but the data will be intact and in this case, the file will be closed only after a Flume restart.
hdfs.retryInterval	180	Time in seconds between consecutive attempts to close a file. Each close call costs multiple RPC round-trips to the Namenode, so setting this too low can cause a lot of load on the name node. If set to 0 or less, the sink will not attempt to close the file if the first attempt fails, and may leave the file open or with a ”.tmp” extension.
serializer	TEXT	Other possible options include avro_event or the fully-qualified class name of an implementation of the EventSerializer.Builder interface.
serializer.*