field [offset] not present as part of path [offset]報錯解決

 

標題是使用filebeat & ingest 進行message字段拆分映射時出現的錯誤。

 

詳細報錯信息

2020-03-24T10:16:13.672+0800    WARN    elasticsearch/client.go:511     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf96793a522be5a1, ext:82629640, loc:(*time.Location)(0x4e5d700)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"eef8f920-0f4d-48a8-a416-70e01917fc85", "hostname":"iZ2ze2rwdd2uyxj2avz8m9Z", "id":"ec322cce-c92a-469b-95c3-ac89af5d730e", "type":"filebeat", "version":"7.5.2"}, "ecs":common.MapStr{"version":"1.1.0"}, "fields":common.MapStr{"type":"access-log"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"iZ2ze2rwdd2uyxj2avz8m9Z", "id":"7e0c05cdccf2bd463a9faca35c78d95c", "name":"iZ2ze2rwdd2uyxj2avz8m9Z", "os":common.MapStr{"codename":"xenial", "family":"debian", "kernel":"4.4.0-142-generic", "name":"Ubuntu", "platform":"ubuntu", "version":"16.04.6 LTS (Xenial Xerus)"}}, "input":common.MapStr{"type":"log"}, "log":common.MapStr{"file":common.MapStr{"path":"/var/log/nginx/access.log"}, "offset":0}, "message":"172.105.205.73 - 82304918473 yuziyue - beijing male at 2018-06-14 14:41:58 \"TimeHut/5.3.3 (iPhone8,2; iOS 9.3.5; Scale/3.00) (SOURCE/App Store, VERSION_CODE/533)\" end"}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000403ba0), Source:"/var/log/nginx/access.log", Offset:167, Timestamp:time.Time{wall:0xbf96793a5228975f, ext:82413022, loc:(*time.Location)(0x4e5d700)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xa222e, Device:0xfd01}}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"illegal_argument_exception","reason":"field [offset] not present as part of path [offset]"}

 

解決

這個是因爲我在elasticsearch中設置ingest的pipeline時將offset字段remove掉了。不remove就可以了。如果跟標題報錯很像，多半是你的pipeline沒有這個字段導致的。

報錯時pipeline如下的定義

PUT  _ingest/pipeline/pipeline-nginx-access
{
  "description" : "nginx access log",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{IP:clientip} - %{DATA:userid} %{DATA:username} - %{DATA:location} %{DATA:sex} at %{DATA:timestamp} \"%{DATA:useragent}\" end"]
      }
    },{
      "geoip":{
        "field": "clientip",
        "target_field": "geoip"
      }
    },{
      "user_agent": {
        "field": "useragent",
        "target_field": "useragent"
      }
    },{
      "date": {
        "field": "timestamp",
        "formats": ["yyyy-MM-dd HH:mm:ss"],
        "target_field": "timestamp"
      }
    },{
      "remove": {
        "field": "offset"
      }
    },{
      "remove": {
        "field": "prospector"
      }
    },{
      "remove": {
        "field": "message"
      }
    }
  ]
}

上面這個remove  offset要去掉。改成如下。

PUT  _ingest/pipeline/pipeline-nginx-access
{
  "description" : "nginx access log",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{IP:clientip} - %{DATA:userid} %{DATA:username} - %{DATA:location} %{DATA:sex} at %{DATA:timestamp} \"%{DATA:useragent}\" end"]
      }
    },{
      "geoip":{
        "field": "clientip",
        "target_field": "geoip"
      }
    },{
      "user_agent": {
        "field": "useragent",
        "target_field": "useragent"
      }
    },{
      "date": {
        "field": "timestamp",
        "formats": ["yyyy-MM-dd HH:mm:ss"],
        "target_field": "timestamp"
      }
    }
  ]
}

filebeat讀取的日誌如下所示

10.95.110.123 - 82304918473 zhangsan - beijing male at 2020-03-14 14:41:58 "TimeHut/5.3.3 (iPhone8,2; iOS 9.3.5; Scale/3.00) (SOURCE/App Store, VERSION_CODE/533)" end
10.95.110.123 - 82304918474 zhangsan - beijing male at 2020-03-14 14:41:58 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36" end

filebeat的配置如下

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  fields:
    type: "access-log"
setup.template.name: "ignore-this"
setup.template.pattern: "ignore-this"
output.elasticsearch:
  hosts: ["localhost:9200"]
  indices:
    - index: "filebeat-nginx-access-%{+yyy.MM.dd}"
      when.equals:
        fields.type: "access-log"
  pipelines:
    - pipeline: "pipeline-nginx-access"
      when.equals:
        fields.type: "access-log"