elasticsearch入門
啓動Elasticsearch：bin/elasticsearch.bat
檢查是否啓動成功：http://localhost:9200/?pretty
服務默認端口 9300 Web 管理平臺端口 9200
將歷史數據從DB中(MYSQL)中最終寫入到ES中有三種方案 ：
第一種：寫程序，鏈接MYSQL，批量的寫入kakfa中，後續在現有邏輯已經完成，可以好low啊 而且麻煩;
第二種：使用kafka的connect 從 mysql 導入 kafka ，kafka的consumer 程序寫入ES中;
第三種:直接從數據庫中寫入 ES中，最直接 最省事。
第三種方案認爲靠譜的實現方式有兩種A 使用ElasticSearch-jdbc 組件;B:使用logstash-jdbc 插件
在ElasticSearch索引中，對應於CRUD中的“創建”和“更新” - 如果對具有給定類型的文檔進行索引，並且要插入原先不存在的ID。 如果具有相同類型和ID的文檔已存在，則會被覆蓋。原文出自【易百教程】，商業轉載請聯繫作者獲得授權，非商業請保留原文鏈接：https://www.yiibai.com/elasticsearch/elasticsearch-getting-start.html
我們對REST API創建一個PUT請求到一個由索引名稱，類型名稱和ID組成的URL。 也就是：http://localhost:9200/<index>/<type>/[<id>]原文出自【易百教程】，商業轉載請聯繫作者獲得授權，非商業請保留原文鏈接：https://www.yiibai.com/elasticsearch/elasticsearch-getting-start.html
索引和類型是必需的，而id部分是可選的。如果不指定ID，ElasticSearch會爲我們生成一個ID。 但是，如果不指定id，應該使用HTTP的POST而不是PUT請求。原文出自【易百教程】，商業轉載請聯繫作者獲得授權，非商業請保留原文鏈接：https://www.yiibai.com/elasticsearch/elasticsearch-getting-start.html
索引名稱是任意的。如果服務器上沒有此名稱的索引，則將使用默認配置來創建一個索引。
至於類型名稱，它也是任意的。 它有幾個用途，包括：每種類型都有自己的ID空間;不同類型具有不同的映射(“模式”，定義屬性/字段應如何編制索引);搜索多種類型是可以的，並且也很常見，但很容易搜索一種或多種指定類型。
更新索引使用與之前完全相同的索引請求,更新後對象中的_version屬性的值會發生變化，版本號(_version)可用於跟蹤文檔已編入索引的次數。它的主要目的是允許樂觀的併發控制
版本號自動添加。
獲取文檔/索引
http://localhost:9200/<index>/<type>/<id> 就可以獲取信息
_search端點
使用_search端點，可選擇使用索引和類型。也就是說，按照以下模式向URL發出請求：<index>/<type>/_search。其中，index和type都是可選的。
指定搜索的字段
查詢字符串查詢有一些可以指定設置，如果不使用，它將會使用默認的設置值。
這樣的設置稱爲“fields”，可用於指定要搜索的字段列表。如果不使用“fields”字段，ElasticSearch查詢將默認自動生成的名爲“_all”的特殊字段，來基於所有文檔中的各個字段匹配搜索。原文出自【易百教程】，商業轉載請聯繫作者獲得授權，非商業請保留原文鏈接：https://www.yiibai.com/elasticsearch/elasticsearch-getting-start.html
curl -XPOST "http://localhost:9200/_search" -d'
{
"query": {
"query_string": {
"query": "ford",
"fields": ["title"]
}
}
}'
條件過濾器
curl -XPOST "http://localhost:9200/_search" -d'
{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "drama"
}
},
"filter": {
"term": { "year": 1962 }
}
}
}
}'
這個兩個命中的數據的 year 字段的值都是等於 1962
無需查詢即可進行過濾
curl -XPOST "http://localhost:9200/_search" -d'
{
"query": {
"filtered": {
"query": {
"match_all": {
}
},
"filter": {
"term": { "year": 1962 }
}
}
}
}'
創建索引
POST http://localhost:9200/schools
響應：{"acknowledged": true}
創建映射和添加數據
POST http://localhost:9200/schools/_bulk
請求體
{
"index":{
"_index":"schools", "_type":"school", "_id":"1"
}
}
{
"name":"Central School", "description":"CBSE Affiliation", "street":"Nagan",
"city":"paprola", "state":"HP", "zip":"176115", "location":[31.8955385, 76.8380405],
"fees":2000, "tags":["Senior Secondary", "beautiful campus"], "rating":"3.5"
}
{
"index":{
"_index":"schools", "_type":"school", "_id":"2"
}
}
{
"name":"Saint Paul School", "description":"ICSE
Afiliation", "street":"Dawarka", "city":"Delhi", "state":"Delhi", "zip":"110075",
"location":[28.5733056, 77.0122136], "fees":5000,
"tags":["Good Faculty", "Great Sports"], "rating":"4.5"
}
{
"index":{"_index":"schools", "_type":"school", "_id":"3"}
}
{
"name":"Crescent School", "description":"State Board Affiliation", "street":"Tonk Road",
"city":"Jaipur", "state":"RJ", "zip":"176114","location":[26.8535922, 75.7923988],
"fees":2500, "tags":["Well equipped labs"], "rating":"4.5"
}
響應結果
{
"took":328, "errors":false,"items":[
{
"index":{
"_index":"schools", "_type":"school", "_id":"1", "_version":1, "_shards":{
"total":2, "successful":1, "failed":0
}, "status":201
}
},
{
"index":{
"_index":"schools", "_type":"school", "_id":"2", "_version":1, "_shards":{
"total":2, "successful":1, "failed":0
}, "status":201
}
},
{
"index":{
"_index":"schools", "_type":"school", "_id":"3", "_version":1, "_shards":{
"total":2, "successful":1, "failed":0
}, "status":201
}
}
]
}
Elasticsearch提供了一個REST API，通過HTTP通過JSON訪問。 Elasticsearch使用以下約定
1.多索引
API中的大多數操作(主要是搜索和其他操作)用於一個或多個索引。 這有助於用戶通過只執行一次查詢來搜索多個位置或所有可用數據。 許多不同的符號用於在多個索引中執行操作。
逗號分隔符號
POST http://localhost:9200/index1,index2,index3/_search
請求體
{
"query":{
"query_string":{
"query":"any_string"
}
}
}
響應：來自index1，index2，index3的JSON對象，其中包含any_string
2.所有索引的_all關鍵字
POST http://localhost:9200/_all/_search
請求體：
{
"query":{
"query_string":{
"query":"any_string"
}
}
}
響應：來自所有索引的JSON對象，並且有any_string
3.通配符(*，+， - )
(其中+表示允許， - 表示不允許)
POST http://localhost:9200/school*/_search
來自所有索引的JSON對象，它們以“school”開頭，但不是schools_gov
4.日期索引名稱中的數學支持
http://localhost:9200/<accountdetail-{now-2d{YYYY.MM.dd|utc}}>/_search
其中：<accountdetail-{now-d}> 對應accountdetail-2016.12.29
<accountdetail-{now-M}> 對應accountdetail-2015.11.30
<accountdetail-{now{YYYY.MM}}> 對應accountdetail-2015.12
美化結果
可以通過附加一個網址查詢參數(即pretty = true)，獲得格式正確的JSON對象的響應
POST http://localhost:9200/schools/_search?pretty = true
索引API
當使用特定映射對相應索引發出請求時，它有助於在索引中添加或更新JSON文檔。 例如，以下請求將JSON對象添加到索引學校和學校映射下。
POST http://localhost:9200/schools/school/4
請求正文
{
"name":"City School", "description":"ICSE", "street":"West End", "city":"Meerut",
"state":"UP", "zip":"250002", "location":[28.9926174, 77.692485], "fees":3500,
"tags":["fully computerized"], "rating":"4.5"
}
響應
{
"_index":"schools", "_type":"school", "_id":"4", "_version":1,
"_shards":{"total":2, "successful":1,"failed":0}, "created":true
}
當請求將JSON對象添加到特定索引時，如果該索引不存在，那麼此API會自動創建該索引以及該特定JSON對象的基礎映射。
可以通過將以下參數的值更改爲false來禁用此功能，這個值是存在於elasticsearch.yml文件中，打開elasticsearch.yml文件設置如下
action.auto_create_index:false
index.mapper.dynamic:false
版本控制
POST http://localhost:9200/schools/school/1?version = 1
請求正文
{
"name":"Central School", "description":"CBSE Affiliation", "street":"Nagan",
"city":"paprola", "state":"HP", "zip":"176115", "location":[31.8955385, 76.8380405],
"fees":2200, "tags":["Senior Secondary", "beautiful campus"], "rating":"3.3"
}
響應內容
{
"_index":"schools", "_type":"school", "_id":"1", "_version":2,
"_shards":{"total":2, "successful":1,"failed":0}, "created":false
}
操作類型
操作類型用於強制創建操作，這有助於避免覆蓋現有文檔
POST http://localhost:9200/tutorials/chapter/1?op_type = create
請求正文
{
"Text":"this is chapter one"
}
響應內容
{
"_index":"tutorials", "_type":"chapter", "_id":"1", "_version":1,
"_shards":{"total":2, "successful":1, "failed":0}, "created":true
}
自動生成ID
當在索引操作中未指定ID時，Elasticsearch自動爲文檔生成ID
獲取API
GET http://localhost:9200/schools/school/1
響應
{
"_index":"schools", "_type":"school", "_id":"1", "_version":2,
"found":true, "_source":{
"name":"Central School", "description":"CBSE Affiliation",
"street":"Nagan", "city":"paprola", "state":"HP", "zip":"176115",
"location":[31.8955385,76.8380405], "fees":2200,
"tags":["Senior Secondary", "beautiful campus"], "rating":"3.3"
}
}
刪除API
通過向Elasticsearch發送HTTP DELETE請求來刪除指定的索引，映射或文檔
DELETE http://localhost:9200/schools/school/4
響應
{
"found":true, "_index":"schools", "_type":"school", "_id":"4", "_version":2,
"_shards":{"total":2, "successful":1, "failed":0}
}
更新API
腳本用於執行此操作，版本控制用於確保在獲取和重建索引期間沒有發生更新
POST http://localhost:9200/schools_gov/school/1/_update
請求正文
{
"script":{
"inline": "ctx._source.fees+ = inc", "params":{
"inc": 500
}
}
}
響應結果
{
"_index":"schools_gov", "_type":"school", "_id":"1", "_version":2,
"_shards":{"total":2, "successful":1, "failed":0}
}
多獲取API
POST http://localhost:9200/_mget
請求正文
{
"docs":[
{
"_index": "schools", "_type": "school", "_id": "1"
},
{
"_index":"schools_gev", "_type":"school", "_id": "2"
}
]
}
響應結果
{
"docs":[
{
"_index":"schools", "_type":"school", "_id":"1",
"_version":1, "found":true, "_source":{
"name":"Central School", "description":"CBSE Afiliation",
"street":"Nagan", "city":"paprola", "state":"HP", "zip":"176115",
"location":[31.8955385,76.8380405], "fees":2000,
"tags":["Senior Secondary", "beatiful campus"], "rating":"3.5"
}
},
{
"_index":"schools_gev", "_type":"school", "_id":"2", "error":{
"root_cause":[{
"type":"index_not_found_exception", "reason":"no such index",
"index":"schools_gev"
}],
"type":"index_not_found_exception", "reason":"no such index",
"index":"schools_gev"
}
}
]
}
多索引
Elasticsearch允許我們搜索存在於所有索引或一些特定索引中的文檔。 例如，如果我們需要搜索名稱包含central的所有文檔。
GET http://localhost:9200/_search?q = name:central
響應
{
"took":78, "timed_out":false, "_shards":{"total":10, "successful":10, "failed":0},
"hits":{
"total":1, "max_score":0.19178301, "hits":[{
"_index":"schools", "_type":"school", "_id":"1", "_score":0.19178301,
"_source":{
"name":"Central School", "description":"CBSE Affiliation",
"street":"Nagan", "city":"paprola", "state":"HP", "zip":"176115",
"location":[31.8955385, 76.8380405], "fees":2000,
"tags":["Senior Secondary", "beautiful campus"], "rating":"3.5"
}
}]
}
}
多類型
還可以在所有類型或某種指定類型的索引中搜索所有文檔。
Get http://localhost:9200/schools/_search?q = tags:sports
響應
{
"took":16, "timed_out":false, "_shards":{"total":5, "successful":5, "failed":0},
"hits":{
"total":1, "max_score":0.5, "hits":[{
"_index":"schools", "_type":"school", "_id":"2", "_score":0.5,
"_source":{
"name":"Saint Paul School", "description":"ICSE Afiliation",
"street":"Dawarka", "city":"Delhi", "state":"Delhi", "zip":"110075",
"location":[28.5733056, 77.0122136], "fees":5000,
"tags":["Good Faculty", "Great Sports"], "rating":"4.5"
}
}]
}
}
刪除索引
DELETE http://localhost:9200/colleges
可以通過使用_all，*刪除所有索引。
在Elasticsearch中，通過使用基於JSON的查詢進行搜索。 查詢由兩個子句組成
1.葉查詢子句 - 這些子句是匹配，項或範圍的，它們在特定字段中查找特定值；
2.複合查詢子句 - 這些查詢是葉查詢子句和其他複合查詢的組合，用於提取所需的信息
匹配所有查詢
POST http://localhost:9200/schools*/_search
請求正文
{
"query":{
"match_all":{}
}
}
響應
{
"took":1, "timed_out":false, "_shards":{"total":10, "successful":10, "failed":0},
"hits":{
"total":5, "max_score":1.0, "hits":[
{
"_index":"schools", "_type":"school", "_id":"2", "_score":1.0,
"_source":{
"name":"Saint Paul School", "description":"ICSE Affiliation",
"street":"Dawarka", "city":"Delhi", "state":"Delhi",
"zip":"110075", "location":[28.5733056, 77.0122136], "fees":5000,
"tags":["Good Faculty", "Great Sports"], "rating":"4.5"
}
},
{
"_index":"schools_gov", "_type":"school", "_id":"2", "_score":1.0,
"_source":{
"name":"Government School", "description":"State Board Affiliation",
"street":"Hinjewadi", "city":"Pune", "state":"MH", "zip":"411057",
"location":[18.599752, 73.6821995], "fees":500, "tags":["Great Sports"],
"rating":"4"
}
},
{
"_index":"schools", "_type":"school", "_id":"1", "_score":1.0,
"_source":{
"name":"Central School", "description":"CBSE Affiliation",
"street":"Nagan", "city":"paprola", "state":"HP",
"zip":"176115", "location":[31.8955385, 76.8380405],
"fees":2200, "tags":["Senior Secondary", "beautiful campus"],
"rating":"3.3"
}
},
{
"_index":"schools_gov", "_type":"school", "_id":"1", "_score":1.0,
"_source":{
"name":"Model School", "description":"CBSE Affiliation",
"street":"silk city", "city":"Hyderabad", "state":"AP",
"zip":"500030", "location":[17.3903703, 78.4752129], "fees":700,
"tags":["Senior Secondary", "beautiful campus"], "rating":"3"
}
},
{
"_index":"schools", "_type":"school", "_id":"3", "_score":1.0,
"_source":{
"name":"Crescent School", "description":"State Board Affiliation",
"street":"Tonk Road", "city":"Jaipur", "state":"RJ", "zip":"176114",
"location":[26.8535922, 75.7923988], "fees":2500,
"tags":["Well equipped labs"], "rating":"4.5"
}
}
]
}
}
匹配查詢
此查詢將文本或短語與一個或多個字段的值匹配。
POST http://localhost:9200/schools*/_search
請求正文
{
"query":{
"match" : {
"city":"pune"
}
}
}
響應
{
"took":1, "timed_out":false, "_shards":{"total":10, "successful":10, "failed":0},
"hits":{
"total":1, "max_score":0.30685282, "hits":[{
"_index":"schools_gov", "_type":"school", "_id":"2", "_score":0.30685282,
"_source":{
"name":"Government School", "description":"State Board Afiliation",
"street":"Hinjewadi", "city":"Pune", "state":"MH", "zip":"411057",
"location":[18.599752, 73.6821995], "fees":500,
"tags":["Great Sports"], "rating":"4"
}
}]
}
}
multi_match查詢
此查詢將文本或短語與多個字段匹配
POST http://localhost:9200/schools*/_search
請求正文
{
"query":{
"multi_match" : {
"query": "hyderabad",
"fields": [ "city", "state" ]
}
}
}
響應
{
"took":16, "timed_out":false, "_shards":{"total":10, "successful":10, "failed":0},
"hits":{
"total":1, "max_score":0.09415865, "hits":[{
"_index":"schools_gov", "_type":"school", "_id":"1", "_score":0.09415865,
"_source":{
"name":"Model School", " description":"CBSE Affiliation",
"street":"silk city", "city":"Hyderabad", "state":"AP",
"zip":"500030", "location":[17.3903703, 78.4752129], "fees":700,
"tags":["Senior Secondary", "beautiful campus"], "rating":"3"
}
}]
}
}
查詢字符串查詢
此查詢使用查詢解析器和query_string關鍵字
POST http://localhost:9200/schools/_search
請求正文
{
"query":{
"query_string":{
"query":"good faculty"
}
}
}
響應
{
"took":16, "timed_out":false, "_shards":{"total":10, "successful":10, "failed":0},
"hits":{
"total":1, "max_score":0.09492774, "hits":[{
"_index":"schools", "_type":"school", "_id":"2", "_score":0.09492774,
"_source":{
"name":"Saint Paul School", "description":"ICSE Affiliation",
"street":"Dawarka", "city":"Delhi", "state":"Delhi",
"zip":"110075", "location":[28.5733056, 77.0122136],
"fees":5000, "tags":["Good Faculty", "Great Sports"],
"rating":"4.5"
}
}]
}
}
期限等級查詢
處理結構化數據，如數字，日期和枚舉
POST http://localhost:9200/schools/_search
請求正文
{
"query":{
"term":{"zip":"176115"}
}
}
響應
{
"took":1, "timed_out":false, "_shards":{"total":10, "successful":10, "failed":0},
"hits":{
"total":1, "max_score":0.30685282, "hits":[{
"_index":"schools", "_type":"school", "_id":"1", "_score":0.30685282,
"_source":{
"name":"Central School", "description":"CBSE Affiliation",
"street":"Nagan", "city":"paprola", "state":"HP", "zip":"176115",
"location":[31.8955385, 76.8380405], "fees":2200,
"tags":["Senior Secondary", "beautiful campus"], "rating":"3.3"
}
}]
}
}
範圍查詢
此查詢用於查找值的範圍之間的值的對象。 爲此，需要使用類似
gte − 大於和等於 gt − 大於 lte − 小於和等於 lt − 小於
POST http://localhost:9200/schools*/_search
請求正文
{
"query":{
"range":{
"rating":{
"gte":3.5
}
}
}
}
響應
{
"took":31, "timed_out":false, "_shards":{"total":10, "successful":10, "failed":0},
"hits":{
"total":3, "max_score":1.0, "hits":[
{
"_index":"schools", "_type":"school", "_id":"2", "_score":1.0,
"_source":{
"name":"Saint Paul School", "description":"ICSE Affiliation",
"street":"Dawarka", "city":"Delhi", "state":"Delhi",
"zip":"110075", "location":[28.5733056, 77.0122136], "fees":5000,
"tags":["Good Faculty", "Great Sports"], "rating":"4.5"
}
},
{
"_index":"schools_gov", "_type":"school", "_id":"2", "_score":1.0,
"_source":{
"name":"Government School", "description":"State Board Affiliation",
"street":"Hinjewadi", "city":"Pune", "state":"MH", "zip":"411057",
"location":[18.599752, 73.6821995] "fees":500,
"tags":["Great Sports"], "rating":"4"
}
},
{
"_index":"schools", "_type":"school", "_id":"3", "_score":1.0,
"_source":{
"name":"Crescent School", "description":"State Board Affiliation",
"street":"Tonk Road", "city":"Jaipur", "state":"RJ", "zip":"176114",
"location":[26.8535922, 75.7923988], "fees":2500,
"tags":["Well equipped labs"], "rating":"4.5"
}
}
]
}
}
複合查詢
這些查詢是通過使用如和，或，非和或等，用於不同索引或具有函數調用等的布爾運算符彼此合併的不同查詢的集合
POST http://localhost:9200/schools*/_search
請求正文
{
"query":{
"filtered":{
"query":{
"match":{
"state":"UP"
}
},
"filter":{
"range":{
"rating":{
"gte":4.0
}
}
}
}
}
}
響應
{
"took":16, "timed_out":false, "_shards":{"total":10, "successful":10, "failed":0},
"hits":{"total":0, "max_score":null, "hits":[]}
}
連接查詢
用於檢索在查詢中匹配的文檔的子文檔或父文檔
POST http://localhost:9200/tutorials/_search
請求正文
{
"query":
{
"has_child" : {
"type" : "article", "query" : {
"match" : {
"Text" : "This is article 1 of chapter 1"
}
}
}
}
}
響應
{
"took":21, "timed_out":false, "_shards":{"total":5, "successful":5, "failed":0},
"hits":{
"total":1, "max_score":1.0, "hits":[{
"_index":"tutorials", "_type":"chapter", "_id":"1", "_score":1.0,
"_source":{
"Text":"this is chapter one"
}
}]
}
}
地理查詢
這些查詢處理地理位置和地理點。這些查詢有助於查找任何位置附近的學校或任何其他地理對象。需要使用地理位置數據類型
POST http://localhost:9200/schools*/_search
請求正文
{
"query":{
"filtered":{
"filter":{
"geo_distance":{
"distance":"100km",
"location":[32.052098, 76.649294]
}
}
}
}
}
響應
{
"took":6, "timed_out":false, "_shards":{"total":10, "successful":10, "failed":0},
"hits":{
"total":2, "max_score":1.0, "hits":[
{
"_index":"schools", "_type":"school", "_id":"2", "_score":1.0,
"_source":{
"name":"Saint Paul School", "description":"ICSE Affiliation",
"street":"Dawarka", "city":"Delhi", "state":"Delhi", "zip":"110075",
"location":[28.5733056, 77.0122136], "fees":5000,
"tags":["Good Faculty", "Great Sports"], "rating":"4.5"
}
},
{
"_index":"schools", "_type":"school", "_id":"1", "_score":1.0,
"_source":{
"name":"Central School", "description":"CBSE Affiliation",
"street":"Nagan", "city":"paprola", "state":"HP", "zip":"176115",
"location":[31.8955385, 76.8380405], "fees":2000,
"tags":["Senior Secondary", "beautiful campus"], "rating":"3.5"
}
}
]
}
}
ES將數據存儲於一個或多個索引中，索引是具有類似特性的文檔的集合。類比傳統的關係型數據庫領域來說，索引相當於SQL中的一個數據庫，或者一個數據存儲方案(schema)。
java關聯elasticsearch的api使用
QueryBuilders.termsQuery(String fieldName, String...value);查詢一個字段的一個或多個對應的值
QueryBuilders.queryStringQuery(req.getCpCharge().toString()).field("isPurchase")查詢某個字段的值，左右模糊
QueryBuilders.termQuery("convergeStatus", req.getConverge().toString())精確匹配字段
QueryBuilders.rangeQuery("albumCreateTime").gt(req.getGitvCreateStart()).lt(req.getGitvCreateEnd());範圍查詢
QueryBuilders.rangeQuery("duration").from(req.getPalyTimeLow()).to(req.getPalyTimeHigh());區間查詢
elasticsearch-head插件安裝
進入elasticsearch2.3.0/bin目錄下，打開cmd,輸入
plugin install mobz/elasticsearch-head
安裝成功後，啓動elasticsearch,輸入網址http://localhost:9200/_plugin/head/
進入到elasticsearch的管理頁面
logstash使用
使用前先啓動elasticsearch及安裝elasticsearch-head
下載解壓後，進入logstash2.3/bin目錄，創建文件logstash.conf，編輯輸入
input {
stdin {
}
}
output {
elasticsearch {hosts => “127.0.0.1:9200” } #elasticsearch服務地址
stdout{
}
}
logstash -f logstash.conf啓動
-f:指定logstash的配置文件
-e:格式是後面跟字符串,這個字符串就被當做是logstash的配置;如果"",那麼默認使用stdin作爲輸出,stdout作爲輸出;
-t:測試配置文件是否正確
,啓動後輸入hello world,進入localhost:9200/_plugin/head可以查看到輸入的數據hello world