EFK6.3+kafka+logstash日誌分析平臺集羣

轉載來源 :EFK6.3+kafka+logstash日誌分析平臺集羣 :https://www.jianshu.com/p/f956ebbb2499

架構解讀 :

第一層、數據採集層
安裝filebeat做日誌採集,同時把採集的日誌發送給kafka broker+zookeeper集羣。

第二層、數據轉發層
Logstash節點會實時去kafka broker集羣拉數據,接受到的日誌經過格式處理,然後轉發到ES集羣

第三層、數據檢索,數據展示
ES Master + Kibana 主要協調ES集羣,處理數據檢索請求,數據展示。
所有服務器環境準備

$ systemctl stop firwalld
$ setenforce 0
$ yum -y install java

1. Elasticsearch集羣服務安裝:

ES各節點創建用戶組

$ groupadd elsearch
$ useradd -g elsearch elsearch
$ chown -R elsearch:elsearch  elasticsearch

設置系統的相關參數,如果不設置參數將會存在相關的問題導致不能啓動

$ vim /etc/security/limits.conf
# End of file
* soft nproc 65535
* hard nproc 65535
* soft nofile 65536
* hard nofile 65536
elsearch soft memlock unlimited
elsearch hard memlock unlimited 

修改最大線程數的配置

$ vim /etc/security/limits.d/20-nproc.conf
 *         soft    nproc     65536 
root       soft    nproc     unlimited
$ vim /etc/sysctl.conf
vm.max_map_count=262144 
fs.file-max=6553
$ sysctl -p

修改配置文件

$ vim /usr/local/elasticsearch/config/elasticsearch.yml 
network.host: 0.0.0.0 (允許所有網段訪問9200)
http.port: 9200

啓動程序

$  su - elsearch
$  /usr/local/elasticsearch/bin/elasticsearch  -d

驗證有沒有啓動成功.

$ curl  192.168.16.221:9200
{
  "name" : "fhASdIt",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "lo_I0yMkTJu0TMl8gCwelw",
  "version" : {
    "number" : "6.3.1",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "eb782d0",
    "build_date" : "2018-06-29T21:59:26.107521Z",
    "build_snapshot" : false,
    "lucene_version" : "7.3.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

將Elasticsearch複製到另外兩臺節點服務器中,只需要更改配置文件即可.
Es-master (192.168.16.221)

 $ cat > elasticsearch.yml <<EOF
# ======================== Elasticsearch Configuration =========================
#集羣的名稱,同一個集羣該值必須設置成相同的
cluster.name: my-cluster
#該節點的名字
node.name: node-1
#該節點有機會成爲master節點
node.master: true
#該節點可以存儲數據
node.data: true
path.data: /usr/local/elasticsearch/data
path.logs: /usr/local/elasticsearch/logs
bootstrap.memory_lock: true
#設置綁定的IP地址,可以是IPV4或者IPV6
network.bind_host: 0.0.0.0
#設置其他節點與該節點交互的IP地址
network.publish_host: 192.168.16.221
#該參數用於同時設置bind_host和publish_host
network.host: 0.0.0.0
#設置節點之間交互的端口號
transport.tcp.port: 9300
#設置是否壓縮tcp上交互傳輸的數據
transport.tcp.compress: true
#設置http內容的最大大小]
http.max_content_length: 100mb
#是否開啓http服務對外提供服務
http.enabled: true
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.16.221:9300","192.168.251:9300", "192.168.16.252:9300"]
discovery.zen.minimum_master_nodes: 2
http.cors.enabled: true
http.cors.allow-origin: "*"
EOF

DataNode01節點(192.168.16.251)

$ cat >elasticsearch.yml  <<EOF
# ======================== Elasticsearch Configuration =========================
#集羣的名稱,同一個集羣該值必須設置成相同的
cluster.name: my-cluster
#該節點的名字
node.name: node-2
#該節點有機會成爲master節點
node.master: true
#該節點可以存儲數據
node.data: true
path.data: /usr/local/elasticsearch/data
path.logs: /usr/local/elasticsearch/logs
bootstrap.memory_lock: true
#設置綁定的IP地址,可以是IPV4或者IPV6
network.bind_host: 0.0.0.0
#設置其他節點與該節點交互的IP地址
network.publish_host: 192.168.16.251
#該參數用於同時設置bind_host和publish_host
network.host: 0.0.0.0
#設置節點之間交互的端口號
transport.tcp.port: 9300
#設置是否壓縮tcp上交互傳輸的數據
transport.tcp.compress: true
#設置http內容的最大大小]
http.max_content_length: 100mb
#是否開啓http服務對外提供服務
http.enabled: true
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.16.221:9300","192.168.16.251:9300", "192.168.16.252:9300"]
discovery.zen.minimum_master_nodes: 2
http.cors.enabled: true
http.cors.allow-origin: "*"
EOF

DataNode02節點(192.168.16.252)

$ cat > elasticsearch.yml <<EOF
# ======================== Elasticsearch Configuration =========================
#集羣的名稱,同一個集羣該值必須設置成相同的
cluster.name: my-cluster
#該節點的名字
node.name: node-3
#該節點有機會成爲master節點
node.master: true
#該節點可以存儲數據
node.data: true
path.data: /usr/local/elasticsearch/data
path.logs: /usr/local/elasticsearch/logs
bootstrap.memory_lock: true
#設置綁定的IP地址,可以是IPV4或者IPV6
network.bind_host: 0.0.0.0
#設置其他節點與該節點交互的IP地址
network.publish_host: 192.168.16.252
#該參數用於同時設置bind_host和publish_host
network.host: 0.0.0.0
#設置節點之間交互的端口號
transport.tcp.port: 9300
#設置是否壓縮tcp上交互傳輸的數據
transport.tcp.compress: true
#設置http內容的最大大小]
http.max_content_length: 100mb
#是否開啓http服務對外提供服務
http.enabled: true
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.16.221:9300","192.168.16.251:9300", "192.168.16.252:9300"]
discovery.zen.minimum_master_nodes: 2
http.cors.enabled: true
http.cors.allow-origin: "*"
EOF

各節點啓動

/usr/local/elasticsearch/bin/elasticsearch  -d

在這裏插入圖片描述

2.在master節點部署kibana

$ ln -s kibana-5.6.2-linux-x86_64 kibana
##修改配置文件
$ vim kibana.yml
server.port: 5601
server.host: "192.168.16.221"
server.name: "Esmaster-Kibana"
elasticsearch.url: http://192.168.16.221:9200
## 啓動kibana
$ nohup  sh /usr/local/kibana/bin/kibana &

訪問kibana
192.168.16.221:5601
在這裏插入圖片描述

3. Zookeeper+Kafka集羣部署:

下載軟件包(需注意版本兼容問題)

$ wget http://mirrors.hust.edu.cn/apache/zookeeper/zookeeper-3.4.10/zookeeper-3.4.10.tar.gz
$ wget http://mirror.bit.edu.cn/apache/kafka/1.1.0/kafka_2.12-1.1.0.tgz

三臺主機hosts如下,必須保持一致.

cat  > /etc/hosts <<EOF  
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.16.222 kafka-01
192.168.16.237 kafka-02
192.168.16.238 kafka-03
EOF

安裝zookeeper
master節點

$ tar -zxvf zookeeper-3.4.10.tar.gz -C /usr/local/
$ cd /usr/local/
$ ln -s zookeeper-3.4.10 zookeeper
$ cd zookeeper/conf/
$ cp zoo_sample.cfg zoo.cfg
$ vim  zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/tmp/zookeeper
clientPort=2181
server.1=kafka-01:2888:3888
server.2=kafka-02:2888:3888
server.3=kafka-03:2888:3888

創建dataDir目錄創建/tmp/zookeeper

在master節點上

$ mkdir /tmp/zookeeper
$ echo 1 > /tmp/zookeeper/myid

將zookeeper文件複製到另外兩個節點:

$  scp -r zookeeper-3.4.10/ kafka-02:/usr/local/
$  scp -r zookeeper-3.4.10/ kafka-03:/usr/local/

在兩個slave節點創建目錄和文件

#ZooKeeper-Kafka-02節點:

 $ ln -s zookeeper-3.4.10 zookeeper
$ mkdir /tmp/zookeeper
$ echo 2 > /tmp/zookeeper/myid

#ZooKeeper-Kafka-03節點

 $ ln -s zookeeper-3.4.10 zookeeper
$  mkdir /tmp/zookeeper
$  echo 3 > /tmp/zookeeper/myid

分別在每個節點上啓動 zookeeper測試:

$ ./bin/zkServer.sh start
$ ./bin/zkServer.sh start
$ ./bin/zkServer.sh start

所有節點啓動後查看狀態:

$ ./bin/zkServer.sh status
ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: follower
$ ./bin/zkServer.sh status
ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: leader
$ ./bin/zkServer.sh status
ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: follower

至此zookeeper集羣安裝完成!

4.Kafka集羣安裝配置

$ tar zxf   kafka_2.12-1.1.0.tgz  -C /usr/local/
$ cd  /usr/local/
$ ln -s kafka_2.12-1.1.0/  kafka
$ cd   kafka/config
$ cat > server.properties <<EOF
broker.id=0
listeners=PLAINTEXT://kafka-01:9092
advertised.listeners=PLAINTEXT://kafka-01:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=5
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=24
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=kafka-01:2181,kafka-02:2181,kafka-03:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
delete.topic.enable=true
EOF

將 kafka_2.12-1.1.0 文件夾複製到另外兩個節點下

$ scp -r  kafka_2.12-1.1.0/    kafka-02:/usr/local/
$ scp -r  kafka_2.12-1.1.0/    kafka-03:/usr/local/

修改每個節點對應的 server.properties 文件的 broker.id和listeners、advertised.listeners的名稱.
kafka-02

broker.id=1
listeners=PLAINTEXT://kafka-02:9092
advertised.listeners=PLAINTEXT://kafka-02:9092
kafka-02

broker.id=2
listeners=PLAINTEXT://kafka-03:9092
advertised.listeners=PLAINTEXT://kafka-03:9092

所有節點執行啓動

$ /usr/local/kafka/bin/kafka-server-start.sh -daemon  /usr/local/kafka/config/server.properties 
查看狀態
$  tail -f /usr/local/kafka/logs/server.log   # 日誌最後一條,顯示已經啓動
...
[2017-12-19 16:10:05,542] INFO [KafkaServer id=3] started (kafka.server.KafkaServer)

創建topic 測試

$  bin/kafka-topics.sh --create --zookeeper kafka-01:2181, kafka-02:2181,kafka-03:2181 --replication-factor 3 --partitions 3 --topic test
 Created topic "test".

顯示topic

$ bin/kafka-topics.sh --describe --zookeeper kafka-01:2181, kafka-02:2181,kafka-03:2181 --topic test
 OpenJDK 64-Bit Server VM warning: If the number of processors is 
 expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
 Topic:test PartitionCount:3    ReplicationFactor:3 Configs:
    Topic: test Partition: 0    Leader: 0   Replicas: 0,1,2 Isr: 0,1,2
    Topic: test Partition: 1    Leader: 1   Replicas: 1,2,0 Isr: 1,2,0
    Topic: test Partition: 2    Leader: 2   Replicas: 2,0,1 Isr: 2,0,1

列出topic

$ bin/kafka-topics.sh --list --zookeeper kafka-01:2181, kafka-02:2181,kafka-03:2181
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
test

創建 producer(生產者);
在master節點上 測試生產消息

$ bin/kafka-console-producer.sh --broker-list kafka-01:9092 -topic test
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
>hello world
>elk 

創建 consumer(消費者):
在kafka-02節點上 測試消費

$ bin/kafka-console-consumer.sh --zookeeper kafka-01:2181,kafka-02:2181,kafka-03:2181 -topic test --from-beginning

Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
hello world
elk 

創建 consumer(消費者):
在kafka-03節點上 測試消費

$ bin/kafka-console-consumer.sh --zookeeper kafka-01:2181,kafka-02:2181,kafka-03:2181 -topic test --from-beginning

Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
hello world
elk 

在 producer 裏輸入消息,consumer 中就會顯示出同樣的內容,表示消費成功!
刪除 topic

$ bin/kafka-topics.sh --delete --zookeeper kafka-01:2181, kafka-02:2181,kafka-03:2181 --topic test

列出topics已經爲空

$ bin/kafka-topics.sh --list --zookeeper kafka-01:2181, kafka-02:2181,kafka-03:2181

啓動和關閉服務:

#啓動服務:
$ /usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties 
#停止服務:
$ bin/kafka-server-stop.sh

5.安裝配置filebeat

filebeat.prospectors:
- input_type: log
encoding: GB2312 #字符集
fields_under_root: true
fields:  ##添加字段
  serverip: 192.168.16.100
  indexname: zam
enabled: True
paths:
      - /app/zamtomcat/logs/catalina.out
multiline.pattern: '^\['  #java報錯過濾
multiline.negate:  true
multiline.match: after
tail_files: false
#----------------------------- Logstash output --------------------------------
output.kafka:
enabled: true
hosts: ["192.168.16.222:9092","192.168.16.237:9092","192.168.16.238:9092"]
topic: "zam-filebeat"
partition.hash:
  reachable_only: true
compression: gzip
max_message_bytes: 1000000
required_acks: 1
logging.to_files: true

啓動filebeat

$ nohup ./filebeat -e -c zam.yml  >/dev/null 2>&1 &

kafka查看是否有 “zam-filebeat” topic

$ bin/kafka-topics.sh --list --zookeeper kafka-01:2181, kafka-02:2181,kafka-03:2181
zam-filebeat

啓動消費者查看是否有數據

 $ bin/kafka-console-consumer.sh --bootstrap-server kafka-01:9092,kafka-02:9092,kafka-03:9092 --topic zam-filebeat --from-beginning
--------------

注意:如沒有數據,需在filebeat主機hosts文件中添加kafka集羣地址解析

6.配置logstash

input{
  kafka{
     bootstrap_servers => "kafka-01:9092,kafka-02:9092,kafka-03:9092"
    topics => "zam-filebeat"
    consumer_threads => 1
    decorate_events => true
    codec => "json"
    auto_offset_reset => "latest"

}
}
filter {
        ruby {
        code => "event.timestamp.time.localtime" ##時區設置
      }


        mutate {
        remove_field => ["beat"] #刪除自帶字段
    }
        grok {
             match => {"message" => "\[(?<time>\d+-\d+-\d+\s\d+:\d+:\d+)\] \[(?<level>\w+)\] (?<thread>[\w|-]+) (?<class>[\w|\.]+) (?<lineNum>\d+):(?<msg>.+)"
 } #正則過濾

}
}
output {
   elasticsearch {
         hosts => ["192.168.16.221:9200","192.168.16.251:9200","192.168.16.252:9200"]
         index =>  "zam-%{+YYYY-MM-dd}"
  }
}

注意:索引名稱不能大寫,注意特殊字符,DNS問題需添加kafka集羣hosts文件解析

啓動 logstash

 $ nohup  ./bin/logstash -f  filebeat.conf    > /dev/null 2>&1 &

在head插件上查看索引
在這裏插入圖片描述
在kibana添加索引
在這裏插入圖片描述
在這裏插入圖片描述
免安裝head插件部署http://www.unmin.club/?p=139
參考:http://www.cnblogs.com/saneri/p/8822116.html

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章