ELK引入redis和引入kafka、zookeeper

原來的架構

這樣的架構會導致ES壓力太大

引入redis架構圖

redis不能直接對接ES 所以使用logstash進行轉換
redis的數據是filebeat輸入的
logstash 從redis（倉庫中）拿取數據給ES

使用redis緩存服務來緩解ES壓力

1.安裝redis
yum install redis 
sed -i 's#^bind 127.0.0.1#bind 127.0.0.1 10.0.0.51#' /etc/redis.conf
systemctl start redis 
netstat -lntup|grep redis 
redis-cli -h 10.0.0.51

2.停止docker容器
docker stop $(docker ps -q)

3.停止filebeat
systemctl stop filebeat 

4.刪除舊的ES索引

5.確認nginx日誌爲json格式
grep "access_log" nginx.conf

6.修改filebeat配置文件
cat >/etc/filebeat/filebeat.yml <<EOF
filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]

- type: log
  enabled: true 
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]

output.redis:
  hosts: ["10.0.0.51"]
  keys:
    - key: "nginx_access"
      when.contains:
        tags: "access"
    - key: "nginx_error"
      when.contains:
        tags: "error"

setup.template.name: "nginx"
setup.template.pattern: "nginx_*"
setup.template.enabled: false
setup.template.overwrite: true
EOF

7.重啓filebaet和nginx
systemctl restart nginx 
systemctl restart filebeat

8.生成測試數據
curl 127.0.0.1/haha

9.檢查
redis-cli -h 10.0.0.51
keys * 
TYPE nginx_access
LLEN nginx_access
LRANGE nginx_access 0 -1 
確認是否爲json格式

10.安裝logstash
rpm -ivh jdk-8u102-linux-x64.rpm 
rpm -ivh logstash-6.6.0.rpm


11.配置logstash
cat >/etc/logstash/conf.d/redis.conf<<EOF 
input {
  redis {
    host => "10.0.0.51"
    port => "6379"
    db => "0"
    key => "nginx_access"
    data_type => "list"
  }
  redis {
    host => "10.0.0.51"
    port => "6379"
    db => "0"
    key => "nginx_error"
    data_type => "list"
  }
}

filter {
  mutate {
    convert => ["upstream_time", "float"]
    convert => ["request_time", "float"]
  }
}

output {
   stdout {}
   if "access" in [tags] {
      elasticsearch {
        hosts => "http://10.0.0.51:9200"
        manage_template => false
        index => "nginx_access-%{+yyyy.MM}"
      }
    }
    if "error" in [tags] {
      elasticsearch {
        hosts => "http://10.0.0.51:9200"
        manage_template => false
        index => "nginx_error-%{+yyyy.MM}"
      }
    }
}
EOF

12.前臺啓動測試
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis.conf 

13.檢查
logstash輸出的內容有沒有解析成json
es-head上有沒有索引生成
redis裏的列表數據有沒有在減少

14.將logstash放在後臺運行
ctrl+c
systemctl start logstash
聽風扇聲音，開始轉的時候表示logstash啓動了  --查看9600端口

redis操作命令

redis-cli -h  10.0.0.51
keys *
type nginx_access
llen nginx_access
lrange nginx_access 0 -1

架構的優化部分

如果現在想要新增加一個日誌文件
需要修改4個地方

redis優化：
新增加一個日誌需要修改4個地方
優化後新增加一個日誌需要修改2個地方
原理：只有在logstash輸出到ES時進行分類前面的filebeat和redis中只是打標籤不需要放到不同的文件中（access error放到一起）

filebeat端口：監控日誌寫到redis中
redis端口：6379
logstash端口：9600
elasticsearch端口： 9200 9300

filebeat 不支持鏈接 redis集羣或者哨兵，所以下圖中 redis 只是一個一個的工作

多個logstash會不會同時讀取redis中的數據（是多個logstash工作）

擴容多臺redis和logstash

1.增加redis備份節點：
cat >/etc/filebeat/filebeat.yml <<'EOF'
filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]

- type: log
  enabled: true 
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]


output.redis:
  hosts: ["10.0.0.51:6379","10.0.0.52:6379"]
  key: "nginx_log"

setup.template.name: "nginx"
setup.template.pattern: "nginx_*"
setup.template.enabled: false
setup.template.overwrite: true
EOF

2.logstash從多臺redis讀取數據：
cat >/etc/logstash/conf.d/redis.conf <<'EOF'
input {
  redis {
    host => "10.0.0.51"
    port => "6379"
    db => "0"
    key => "nginx_log"
    data_type => "list"
  }
  redis {
    host => "10.0.0.52"
    port => "6379"
    db => "0"
    key => "nginx_log"
    data_type => "list"
  }
}

filter {
  mutate {
    convert => ["upstream_time", "float"]
    convert => ["request_time", "float"]
  }
}

output {
   stdout {}
   if "access" in [tags] {
      elasticsearch {
        hosts => "http://10.0.0.51:9200"
        manage_template => false
        index => "nginx_access-%{+yyyy.MM}"
      }
    }
    if "error" in [tags] {
      elasticsearch {
        hosts => "http://10.0.0.51:9200"
        manage_template => false
        index => "nginx_error-%{+yyyy.MM}"
      }
    }
}
EOF

這個架構圖中 logstash 讀取redis是輪詢的

消息隊列：
kafka 、zookeeper 配套使用

原理圖：
這樣的架構比使用redis更大性能更高，一天100G之前的

生產者放消息消費者拿消息

kafka和zookeeper 引入

0.配置密鑰
cat >/etc/hosts<<EOF
10.0.0.51 db01
10.0.0.52 db02
10.0.0.53 db03
EOF
ssh-keygen
ssh-copy-id 10.0.0.52
ssh-copy-id 10.0.0.53

1.安裝zook
###db01操作
cd /data/soft
tar zxf zookeeper-3.4.11.tar.gz -C /opt/
ln -s /opt/zookeeper-3.4.11/ /opt/zookeeper
mkdir -p /data/zookeeper
cat >/opt/zookeeper/conf/zoo.cfg<<EOF
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper
clientPort=2181
server.1=10.0.0.51:2888:3888
server.2=10.0.0.52:2888:3888
server.3=10.0.0.53:2888:3888
EOF
echo "1" > /data/zookeeper/myid
cat /data/zookeeper/myid
rsync -avz /opt/zookeeper* 10.0.0.52:/opt/
rsync -avz /opt/zookeeper* 10.0.0.53:/opt/

###db02操作
mkdir -p /data/zookeeper
echo "2" > /data/zookeeper/myid
cat /data/zookeeper/myid

###db03操作
mkdir -p /data/zookeeper
echo "3" > /data/zookeeper/myid
cat /data/zookeeper/myid

2.啓動zookeeper
/opt/zookeeper/bin/zkServer.sh start

3.檢查啓動是否成功
/opt/zookeeper/bin/zkServer.sh status
如果啓動正常mode應該是
2個follower
1個leader

4.測試zook通訊是否正常
在一個節點上執行,創建一個頻道
/opt/zookeeper/bin/zkCli.sh -server 10.0.0.51:2181
create /test "hello"

在其他節點上看能否接收到
/opt/zookeeper/bin/zkCli.sh -server 10.0.0.52:2181
get /test

5.安裝kafka
###db01操作
cd /data/soft/
tar zxf kafka_2.11-1.0.0.tgz -C /opt/
ln -s /opt/kafka_2.11-1.0.0/ /opt/kafka
mkdir /opt/kafka/logs
cat >/opt/kafka/config/server.properties<<EOF
broker.id=1
listeners=PLAINTEXT://10.0.0.51:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/opt/kafka/logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=24
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=10.0.0.51:2181,10.0.0.52:2181,10.0.0.53:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
EOF
rsync -avz /opt/kafka* 10.0.0.52:/opt/
rsync -avz /opt/kafka* 10.0.0.53:/opt/


###db02操作
sed -i "s#10.0.0.51:9092#10.0.0.52:9092#g" /opt/kafka/config/server.properties
sed -i "s#broker.id=1#broker.id=2#g" /opt/kafka/config/server.properties

###db03操作
sed -i "s#10.0.0.51:9092#10.0.0.53:9092#g" /opt/kafka/config/server.properties
sed -i "s#broker.id=1#broker.id=3#g" /opt/kafka/config/server.properties


6.先前臺啓動kafka測試
/opt/kafka/bin/kafka-server-start.sh  /opt/kafka/config/server.properties

7.檢查是否啓動
jps

8.kafka測試命令發送消息
創建命令
/opt/kafka/bin/kafka-topics.sh --create --zookeeper 10.0.0.51:2181,10.0.0.52:2181,10.0.0.53:2181 --partitions 3 --replication-factor 3 --topic  messagetest

測試獲取所有的頻道
/opt/kafka/bin/kafka-topics.sh  --list --zookeeper 10.0.0.51:2181,10.0.0.52:2181,10.0.0.53:2181

測試發送消息
/opt/kafka/bin/kafka-console-producer.sh --broker-list  10.0.0.51:9092,10.0.0.52:9092,10.0.0.53:9092 --topic  messagetest

其他節點測試接收
/opt/kafka/bin/kafka-console-consumer.sh --zookeeper 10.0.0.51:2181,10.0.0.52:2181,10.0.0.53:2181 --topic messagetest --from-beginning

9.測試成功之後,可以放在後臺啓動
按ctrl + c 停止kafka的前臺啓動，切換到後臺啓動
/opt/kafka/bin/kafka-server-start.sh  -daemon /opt/kafka/config/server.properties

10.配置filebeat
cat >/etc/filebeat/filebeat.yml <<EOF
filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]

- type: log
  enabled: true 
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]

output.kafka:
  hosts: ["10.0.0.51:9092", "10.0.0.52:9092", "10.0.0.53:9092"]
  topic: 'filebeat'

setup.template.name: "nginx"
setup.template.pattern: "nginx_*"
setup.template.enabled: false
setup.template.overwrite: true
EOF

重啓filebeat
systemctl restart filebeat 

11.訪問並檢查kafka裏有沒有收到日誌
curl 10.0.0.51

/opt/kafka/bin/kafka-topics.sh  --list --zookeeper 10.0.0.51:2181,10.0.0.52:2181,10.0.0.53:2181

/opt/kafka/bin/kafka-console-consumer.sh --zookeeper 10.0.0.51:2181,10.0.0.52:2181,10.0.0.53:2181 --topic filebeat --from-beginning


12.logstash配置文件
cat > /etc/logstash/conf.d/kafka.conf<<EOF 
input {
  kafka{
    bootstrap_servers=>["10.0.0.51:9092,10.0.0.52:9092,10.0.0.53:9092"]
    topics=>["filebeat"]
    group_id=>"logstash"
    codec => "json"
  }
}

filter {
  mutate {
    convert => ["upstream_time", "float"]
    convert => ["request_time", "float"]
  }
}

output {
   stdout {}
   if "access" in [tags] {
      elasticsearch {
        hosts => "http://10.0.0.51:9200"
        manage_template => false
        index => "nginx_access-%{+yyyy.MM}"
      }
    }
    if "error" in [tags] {
      elasticsearch {
        hosts => "http://10.0.0.51:9200"
        manage_template => false
        index => "nginx_error-%{+yyyy.MM}"
      }
    }
}
EOF

13.前臺啓動logatash測試
先清空ES以前生成的索引
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/kafka.conf 

生成訪問日誌
curl 127.0.0.1


14.總結kafka實驗
1.前提條件
- kafka和zook都是基於java的，所以需要java環境
- 這倆比較喫資源，內存得夠

2.安裝zook注意
- 每臺機器的myid要不一樣，而且要和配置文件裏的id對應上
- 啓動測試，角色爲leader和follower
- 測試發送和接受消息 

3.安裝kafka注意
- kafka依賴於zook，所以如果zook不正常,kafka不能工作
- kafka配置文件裏要配上zook的所有IP的列表
- kafka配置文件裏要注意，寫自己的IP地址
- kafka配置文件裏要注意，自己的ID是zook裏配置的myid
- kafka啓動要看日誌出現started纔算是成功

4.測試zook和kafka
- 一端發送消息
- 兩端能實時接收消息

5.配置filebeat
- output要配上kafka的所有的IP列表

6.配置logstash
- input要寫上所有的kafka的IP列表，別忘了[]
- 前臺啓動測試成功後再後臺啓動

7.毀滅測試結果
- 只要還有1個zook和1個kafka節點，就能正常收集日誌

ELK引入redis和引入kafka、zookeeper

詳解redis5.x版本

只使用mongo作爲數據存儲的DoraCMS系統安裝部署

tomcat詳細介紹

Git命令無法補全問題

LAMP搭建owncloud 私有網盤

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結