LogStash 從mysql向es導數據 date_time_parse_exception could not be parsed at index

1、Es 索引類型中有個字段 permissiondate  date類型 format :yyyy-mm-dd

 

2、mysql 中permission_date 字段爲 date類型 ,例:2021-05-02

 

3、通過 LogStash 從mysql 向 Es 中導數據時,報如下錯誤

 

[2022-09-14T14:15:04,827][WARN ][logstash.outputs.elasticsearch][main][90de522c9959b03fa6af43a808fd4e974b06a7c80c9f43b252e9b57c7d9959cf] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"patentinfo", :routing=>nil}, {"patenteecity"=>"賴旁", "@version"=>"1", "abstracts"=>"Multi-layered demand-driven complexity", "publicno"=>"CNutilize visionary vortals", "permissiondate"=>1982-12-19T16:00:00.000Z, "patenteeprovince"=>"馬鞍山", "applicationdate"=>2002-12-18T16:00:00.000Z, "applicationno"=>"CN1104460", "publicdate"=>1964-07-22T16:00:00.000Z, "locarno"=>"94-64", "title"=>"Clouds of Witness", "@timestamp"=>2022-09-14T06:15:03.108Z, "agency"=>"朱, 方 and 金", "patentee"=>"曾峻熙", "locarnosubdivision"=>"33-46-65", "permissionno"=>"CN699037.0", "imgpath"=>nil, "agent"=>"建輝.潘", "patenteeaddress"=>"Apt. 850 曹街1號, 上海, 豫 723706", "designer"=>"蘇浩然", "pdfpath"=>nil}], :response=>{"index"=>{"_index"=>"patentinfo", "_type"=>"_doc", "_id"=>"Iz-jOoMBQcCVuGudAPtx", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [permissiondate] of type [date] in document with id 'Iz-jOoMBQcCVuGudAPtx'. Preview of field's value: '1982-12-19T16:00:00.000Z'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [1982-12-19T16:00:00.000Z] with format [yyyy-MM-dd]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"date_time_parse_exception: Text '1982-12-19T16:00:00.000Z' could not be parsed, unparsed text found at index 10"}}}}}}

 

4、原因:mysql 中的數據格式爲  1982-12-19

               通過LogStash導數據時 格式變爲 1982-12-19T16:00:00.000Z

               而Es索引類型要求的格式爲 :yyyy-MM-dd

               所以報錯

 

5、處理方法:在LogStash jdbc.sql 中 寫的查詢sql時,增加 Data_Format 操作

     如:SELECT
DATE_FORMAT(publicdate,'%Y-%m-%d %H:%i:%S') AS publicdate,
DATE_FORMAT(applicationdate,'%Y-%m-%d %H:%i:%S') AS applicationdate,
DATE_FORMAT(permissiondate,'%Y-%m-%d %H:%i:%S') AS permissiondate
FROM
table

 

處理好後,重新運行LogStash腳本,一切正常

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章