ELK分析MySQL慢查詢日誌並生成圖像

一、背景

1.MySQL慢查詢日誌格式：

# Time: 181109 15:04:08
# User@Host: t***[t***] @  [172.16.14.51]  Id: 8960747
# Query_time: 35.918265  Lock_time: 0.000141 Rows_sent: 1  Rows_examined: 11699162
SET timestamp=1541747048;
select count(*) from trade_risk_control_record

2.MySQL慢查詢日誌已通過rsyslog實時傳輸到logstash作爲Indexer的節點。

二、logstash配置文件

input部分

input {
  file {
    type => "logstash-rc-mysql-slow"
        path => "/opt/data/logs/localhost-172.16.14.35/db1-slow.log"
        codec => multiline {
          pattern => "^# Time:"
          negate => true
          what => "previous"
        }
        stat_interval => 1
                discover_interval => 1
                start_position=>"end"
                sincedb_path => "/dev/null"
  }
}

filter部分

if [type] == "logstash-rc-mysql-slow" {
  grok {
    patterns_dir => ["/usr/local/logstash/etc/conf.d/patterns/mysql"]
    match => { "message" => "%{LONGQUERYLOG}" }
  }
  date {
    match => ["timestamp","UNIX"]
  }
  mutate {
    convert => [ "query_time", "float" ]
    convert => [ " lock_time", "float" ]
    remove_field => "message"
    remove_field => "timestamp"
       }
  } 

output部分

if [type] == "logstash-rc-mysql-slow" {
             elasticsearch {
                            hosts => ["172.16.1.25","172.16.1.26","172.16.1.27"]
                            index => 'logstash-mysql_slow_log-%{+YYYY-MM-dd}'
                            codec=>plain{charset=>"UTF-8"}
                          }
        }

patterns部分

LONGQUERYLOG ^#\s+Time:.*\n#\s+User@Host:\s+%{USER:user}\[[^\]]+\]\s+@\s+(?:(?<clienthost>\S*) )?\[(?:%{IP:clientip})?\]\s+Id:\s+%{NUMBER:id}\n# Query_time: %{NUMBER:query_time}\s+Lock_time: %{NUMBER:lock_time}\s+Rows_sent: %{NUMBER:rows_sent}\s+Rows_examined: %{NUMBER:rows_examined}\nSET\s+timestamp=%{NUMBER:timestamp};\n(?<query>[\s\S]*)

三、kibana展示

1.創建索引

2.發現數據

包括字段：

3.繪製visualize
例1：統計數量排名前10的sql語句及對應的查詢時間