CDH相關的軟件下載地址：http://archive.cloudera.com/cdh5/cdh/5/
cdh-5.7.0
生產或者測試環境選擇對應CDH版本時，一定要採用尾號是一樣的版本
下載flume：
lunx命令： wget http://archive.cloudera.com/cdh5/cdh/5/ flume-ng-1.6.0-cdh5.7.0.tar.gz
配置conf
[root@Machine5 apache-flume-1.6.0-cdh5.7.0-bin]# cd conf/
[root@Machine5 conf]# cp flume-env.sh.template flume-env.sh
[root@Machine5 conf]# vim flume-env.sh
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be sourced
# during Flume startup.
# Enviroment variables can be set here.
export JAVA_HOME=/opt/jdk1.8.0_191
# Give Flume more memory and pre-allocate, enable remote monitoring via JMX
# export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"
# Note that the Flume conf directory is always included in the classpath.
#FLUME_CLASSPATH=""
修改flume-env.sh主要是配置export JAVA_HOME=/opt/jdk1.8.0_191
參考官網： http://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html#avro-source
example.conf
# example.conf: A single-node Flume configuration
# Name the components on this agen
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
#a1.channels.c1.capacity = 1000
#a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
一個終端：
啓動命令：
[root@Machine5 apache-flume-1.6.0-cdh5.7.0-bin]# bin/flume-ng agent --conf conf --conf-file conf/example.conf --name a1 -Dflume.root.logger=INFO,console
另一個終端：
[root@Machine5 conf]# telnet localhost 44444
結果顯示：
日誌收集過程（跨機器）：
exec-memory-avro.conf
# Name the components on this agent
exec-memory-avro.sources = r1
exec-memory-avro.sinks = k1
exec-memory-avro.channels = c1
# Describe/configure the source
exec-memory-avro.sources.r1.type = exec
exec-memory-avro.sources.r1.command = tail -F /root/data.log
exec-memory-avro.sources.tailsource-1.shell = /bin/bash -c
# Describe the sink
exec-memory-avro.sinks.k1.type = avro
exec-memory-avro.sinks.k1.hostname = 127.0.0.1
exec-memory-avro.sinks.k1.port = 4545
# Use a channel which buffers events in memory
exec-memory-avro.channels.c1.type = memory
# Bind the source and sink to the channel
exec-memory-avro.sources.r1.channels = c1
exec-memory-avro.sinks.k1.channel = c1
啓動命令：
bin/flume-ng agent --conf conf --conf-file conf/exec-memory-avro.conf --name exec-memory-avro -Dflume.root.logger=INFO,console
avro-memory-logger.conf
avro-memory-logger.sources = r1
avro-memory-logger.sinks = k1
avro-memory-logger.channels = c1
avro-memory-logger.sources.r1.type =avro
avro-memory-logger.sources.r1.bind = 127.0.0.1
avro-memory-logger.sources.r1.port = 4545
avro-memory-logger.sinks.k1.type = logger
avro-memory-logger.channels.c1.type = memory
# Bind the source and sink to the channel
avro-memory-logger.sources.r1.channels = c1
avro-memory-logger.sinks.k1.channel = c1
啓動命令：
bin/flume-ng agent --conf conf --conf-file conf/avro-memory-logger.conf --name avro-memory-logger -Dflume.root.logger=INFO,console
先啓動 avro-memory-logger.conf 在啓動 exec-memory-avro.conf
運行結果：
啓動 avro-memory-logger.conf的終端會顯示如下：
