相信很多探索ELK的朋友和我一樣，總是想把nginx訪問日誌的索引名稱修改爲自己想要的名稱模式，
例如：nginx-access-YY.MM.DD，不相信使用默認的必須以logstash-開頭的，但是就這一個更改卻
可能帶來很多的問題，比較常見的是自定義的映射模板導入失敗，參數不生效，geoip的定位信息無法
在kibana中調用，筆者也是吃盡了苦頭，查閱了很多的技術博客，理解了模板映射的原理後，反覆嘗試
才成功使用上了自定義的模板映射文件。不知道很多前輩是踩過坑都不說還是之前的版本有新版有區別，
反正沒有看到對此問題說的特別清晰的文章，所以筆者吃盡苦頭之後，還是把自己的心路歷程寫出來，
希望對後人有所幫助。
cat /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/elasticsearch-template-es7x.json
{
"template" : "logstash-*",
"version" : 60001,
"settings" : {
"index.refresh_interval" : "5s",
"number_of_shards": 1
},
"mappings" : {
"_doc" : {
"dynamic_templates" : [ {
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false
}
}
}, {
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text", "norms" : false,
"fields" : {
"keyword" : { "type": "keyword", "ignore_above": 256 }
}
}
}
} ],
"properties" : {
"@timestamp": { "type": "date"},
"@version": { "type": "keyword"},
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "half_float" },
"longitude" : { "type" : "half_float" }
}
}
}
}
}
}
如何直接複製以上內容，修改索引名稱後使用kibana提供的ES的控制檯導入會報如下錯誤：
#! Deprecation: Deprecated field [template] used, replaced by [index_patterns]
{
"acknowledged": true
}
可以導入成功，那是因爲ES會自我修正，把"template" : "nginx-*"修改爲
"index_patterns" : ["nginx-*"]
所以正確的自定義義映射模板文件內容應該如下：
{
"index_patterns" : ["nginx-*"],
"version" : 60001,
"settings" : {
"index.refresh_interval" : "5s",
"number_of_shards": 1
},
"mappings" : {
"_doc" : {
"dynamic_templates" : [ {
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false
}
}
}, {
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text", "norms" : false,
"fields" : {
"keyword" : { "type": "keyword", "ignore_above": 2048 }
}
}
}
} ],
"properties" : {
"@timestamp": { "type": "date"},
"@version": { "type": "keyword"},
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "half_float" },
"longitude" : { "type" : "half_float" }
}
}
}
}
}
}
如果你是第一時間就使用這種方法，筆者只能說你非常走運，可能兩步就能擺平這個自定義映射模板的問題。
但我相信很多人的遭遇和筆者一樣，走了很多的彎路，筆者當時就一直想使用logstash來自己管理這個映射
模板文件，但經過筆者多次嘗試，如果直接使用curl 127.0.0.1:9200/_template/logstash?pretty導出
並重定向到一個文件，再修改的話就遇上大坑了。
curl 127.0.0.1:9200/_template/logstash?pretty > nginx.json
vim nginx.json
{
"nginx" : {
"order" : 0,
"version" : 60001,
"index_patterns" : [
"nginx-*"
],
"settings" : {
"index" : {
"refresh_interval" : "5s"
}
},
"mappings" : {
"_default_" : {
"dynamic_templates" : [
{
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false
}
}
},
{
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 1024
}
}
}
}
}
],
"properties" : {
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "keyword"
},
"geoip" : {
"dynamic" : true,
"properties" : {
"ip" : {
"type" : "ip"
},
"location" : {
"type" : "geo_point"
},
"latitude" : {
"type" : "half_float"
},
"longitude" : {
"type" : "half_float"
}
}
}
}
}
},
"aliases" : { }
}
}
此時在logstash的配置文件中使用如下配置，啓動logstash後，可以看到logstash會報錯
elasticsearch {
hosts => ["192.168.10.101:9200"]
index => "nginx-%{+YYYY.MM.dd}"
template => "/etc/logstash/nginx.json"
template_name => "nginx"
template_overwrite => true
}
logstash日誌報錯如下：
[2018-09-17T04:43:46,342][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/nginx
[2018-09-17T04:43:46,504][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at
URL 'http://192.168.10.101:9200/_template/nginx'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>["/usr/shar
e/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in
`perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_clie
nt/pool.rb:291:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/output
s/elasticsearch/http_client/pool.rb:278:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2
.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output
-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/ge
ms/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:in `block in Pool'", "/usr/share/logstash/vendor/bundl
e/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:348:in `template_put'", "/usr/share/logstash/ve
ndor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:86:in `template_install'", "/usr/shar
e/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:21:in `install'", "
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:9:in `inst
all_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/common.rb:118
:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/comm
on.rb:49:in `block in install_template_after_successful_connection'"]}
如果複製nginx.json的文件內容直接在kibana的ES控制檯上導入的話，會直接看到報錯提示
PUT _template/nginx-test
{
"nginx" : {
"order" : 0,
"version" : 60001,
"index_patterns" : [
"nginx-*"
],
"settings" : {
"index" : {
"refresh_interval" : "5s"
}
},
"mappings" : {
"_default_" : {
"dynamic_templates" : [
{
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false
}
}
},
{
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 1024
}
}
}
}
}
],
"properties" : {
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "keyword"
},
"geoip" : {
"dynamic" : true,
"properties" : {
"ip" : {
"type" : "ip"
},
"location" : {
"type" : "geo_point"
},
"latitude" : {
"type" : "half_float"
},
"longitude" : {
"type" : "half_float"
}
}
}
}
}
},
"aliases" : { }
}
}
右邊返回信息欄中報錯如下：
{
"error": {
"root_cause": [
{
"type": "action_request_validation_exception",
"reason": "Validation Failed: 1: index patterns are missing;"
}
],
"type": "action_request_validation_exception",
"reason": "Validation Failed: 1: index patterns are missing;"
},
"status": 400
}
當初筆者就是爲此報錯抓狂過，但最後算是繞了一個很大的圈子找到了第一種方式的文件模板，最後才成功實現
給nginx自定義映射模板。所以，如果正在抓狂的你看到了我的這篇文章，是不是有一種雪中送炭的感覺了！，
如果感覺有，請點贊+轉發！