logstash json 格式日誌timestamp覆蓋寫入elasticsearch

我們的需求是通過logstash將json格式的log導入到elasticsearch並且使用log中的記錄時間覆蓋@timestamp字段
最簡單的方式是在json中將時間字段改直接命名爲@timestamp，如下json格式示例：

{"@timestamp":"2018-05-08T08:20:40.644+08:00","source":"ap","ip":"192.168.1.1"}

這裏要注意時間的格式爲ISO8601標準，否則elasticsearch無法正確解析

創建logstash配置文件json.conf如下:

input {
    file {
        type => "log"
        path => "/root/logs/*.log"
        start_position => "beginning"
    }
}


filter {
    json{
        source => "message"
    }
}

output {
    stdout {
        codec => rubydebug { }
    }

    elasticsearch {
        hosts => "127.0.0.1"
        index => "logstash-%{type}-%{+YYYY.MM.dd}"
    }
}

啓動logstash：

logstash  -f json.conf

輸出如下：

{
    "@timestamp" => 2018-05-08T00:20:40.644Z,
        "source" => "ap",
          "type" => "log",
      "@version" => "1",
          "path" => "/root/logs/test.log",
          "host" => "centos",
            "ip" => "192.168.1.1",
       "message" => "{\"@timestamp\":\"2018-05-08T08:20:40.644+08:00\",\"source\":\"ap\",\"ip\":\"192.168.1.1\"}"
}

看下@timestamp字段時間確實是我們log裏記錄的時間（elasticsearch中以UTC時區顯示出來以Z結尾標識，所以和我們北京所在的時區數值上相差8個小時）

如果你的json裏不想以@timestamp命名時間可以在filter裏增加date過濾來處理
如下json格式示例：

{"isotime":"2018-05-08T08:20:40.644+08:00","source":"ap","ip":"192.168.1.1"}

創建logstash配置文件json2.conf如下:

input {
    file {
        type => "log"
        path => "/root/logs/*.log"
        start_position => "beginning"
    }
}


filter {
    json{
        source => "message"
    }
    date {
        match => ["isotime", "ISO8601"]
        target => "@timestamp"
    }
}

output {
    stdout {
        codec => rubydebug { }
    }

    elasticsearch {
        hosts => "127.0.0.1"
        index => "logstash-%{type}-%{+YYYY.MM.dd}"
    }
}

啓動logstash：

logstash  -f json2.conf

輸出如下：

{
      "@version" => "1",
       "isotime" => "2018-05-08T08:20:40.644+08:00",
        "source" => "ap",
          "host" => "centos",
    "@timestamp" => 2018-05-08T00:20:40.644Z,
          "type" => "log",
            "ip" => "192.168.1.1",
       "message" => "{\"isotime\":\"2018-05-08T08:20:40.644+08:00\",\"source\":\"ap\",\"ip\":\"192.168.1.1\"}",
          "path" => "/root/logs/test.log"
}

可以在date過濾器下增加remove_field =>[“isotime”]配置將重複的isotime字段去除

建議在測試調試時輸入輸出使用如下形式的配置，直接在命令行輸入輸出看到結果

input { stdin { } }

filter {
    json{
        source => "message"
    }
}

output { stdout { codec => rubydebug } }