filebeat+logstash收集postgres慢查詢日誌

背景：在 win10下docker啓動postgres 、python連接postgres，創建表和插入數據 以及ELK+Filebeat的基礎上進行

一、Filebeat端處理

1. 配置文件修改

1.1 filebeat.yml配置文件新增數據源類型

- type: log
  enabled: true
  paths:
    - /usr/share/filebeat/log/facilities/*/*.log
  tags: ["facility"]
  fields:
    env: test
    source: facility
    type: logs
  fields_under_root: true
  multiline.pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}'
  multiline.negate: true
  multiline.match: after
  include_lines: [ 'duration' ]

 1.2 logstash.conf配置文件中新增過濾類型處理

if [source] == "facility" {
        grok {
            match => [ "message", "(?m)^%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{TZ:timezone}%{SPACE}\[%{POSINT:pid}\]%{SPACE}%{WORD:logLevel}:%{SPACE}duration:%{SPACE}%{NUMBER:duration}%{SPACE}ms%{SPACE}statement:%{SPACE}%{GREEDYDATA:message}" ]
            overwrite => [ "message" ]
        }
    }

1.3 my-postgres.conf 文件在默認配置基礎上新增配置，開啓日誌文件慢查詢記錄

logging_collector = on
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'
log_directory = '/var/log/postgresql'
log_min_duration_statement = 1200	# milliseconds,記錄執行1.2秒及以上的語句

1.4 docker-compose-postgres.yml文件修改默認日誌掛載路徑 

- /d/usr/share/postgresql/log:/var/log/postgresql
修改爲
- /d/usr/share/filebeat/facilities/log/postgresql:/var/log/postgresql

 2. 重啓容器

2.1 重啓postgres、filebeat、logstash容器

docker restart postgres filebeat logstash

3. 生成慢查詢日誌

3.1 連接postgres數據庫，執行查詢語句

4. 查看結果

4.1 日誌文件內容

4.2 日誌收集結果 

 

二、Logstash端處理

1. 配置文件修改

1.1 filebeat.yml配置文件註釋掉include_lines行

# include_lines: [ 'duration' ]

1.2 logstash.conf配置文件過濾部分修改爲

    if [source] == "facility" {
        if "duration" in [message] {
            grok {
                match => [ "message", "(?m)^%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{TZ:timezone}%{SPACE}\[%{POSINT:pid}\]%{SPACE}%{WORD:logLevel}:%{SPACE}duration:%{SPACE}%{NUMBER:duration}%{SPACE}ms%{SPACE}statement:%{SPACE}%{GREEDYDATA:message}" ]
                overwrite => [ "message" ]
            }
        } else {
            # 刪除不匹配的行
            drop {}
        }
    }

2. 重啓容器

2.1 重啓filebeat、logstash容器

docker restart filebeat logstash

3. 追加日誌

3.1 用文本編輯器直接在原日誌文件內追加日誌內容

2020-05-29 08:35:21.546 UTC [26] LOG:  redo done at 2/26326390
2020-05-29 08:35:22.214 UTC [1] LOG:  database system is ready to accept connections
2020-05-29 08:36:19.817 UTC [33] LOG:  duration: 33071.529 ms  statement: SELECT * FROM test.public.company where address = '深圳'
2020-05-29 08:36:19.817 UTC [33] LOG:  duration: 33071.529 ms  statement: SELECT * FROM test.public.company where address = '北京'
2020-05-29 08:35:22.214 UTC [1] LOG:  database system is ready to accept connections

4. 查看結果

4.1 日誌收集結果