ElasticSearch簡單使用

What is ElasticSearch ？

Elasticsearch是一個基於Apache Lucene(TM)的開源搜索引擎。無論在開源還是專有領域，Lucene可以被認爲是迄今爲止最先進、性能最好的、功能最全的搜索引擎庫。

但是，Lucene只是一個庫。想要使用它，你必須使用Java來作爲開發語言並將其直接集成到你的應用中，更糟糕的是，Lucene非常複雜，你需要深入瞭解檢索的相關知識來理解它是如何工作的。

Elasticsearch也使用Java開發並使用Lucene作爲其核心來實現所有索引和搜索的功能，但是它的目的是通過簡單的RESTful API來隱藏Lucene的複雜性，從而讓全文搜索變得簡單。

不過，Elasticsearch不僅僅是Lucene和全文搜索，我們還能這樣去描述它：

分佈式的實時文件存儲，每個字段都被索引並可被搜索
分佈式的實時分析搜索引擎
可以擴展到上百臺服務器，處理PB級結構化或非結構化數據

而且，所有的這些功能被集成到一個服務裏面，你的應用可以通過簡單的RESTful API、各種語言的客戶端甚至命令行與之交互。

上手Elasticsearch非常容易。它提供了許多合理的缺省值，並對初學者隱藏了複雜的搜索引擎理論。它開箱即用（安裝即可使用），只需很少的學習既可在生產環境中使用。

Elasticsearch在Apache 2 license下許可使用，可以免費下載、使用和修改。

隨着你對Elasticsearch的理解加深，你可以根據不同的問題領域定製Elasticsearch的高級特性，這一切都是可配置的，並且配置非常靈活。

Config elasticSearch.properties

#配置elasticsearch index
csg_bill_index_name=hsc_test_index
#配置elasticsearch type
csg_bill_type_name=billdetails
#配置elasticsearch es address 
csg_bill_es_address=l-test.h.beta.cn0:9300
#配置elasticsearch cluster
csg_bill_es_cluster=elasticsearch_f

Create elasticSearch TransportSessionFactory

/**
 * Created by xueping.you on 15-7-30.
 */
@Service
public class ESTransportSessionFactory  {

    private final static Logger LOGGER = LoggerFactory.getLogger(ESTransportSessionFactory.class);

    @Value("${csg_bill_es_address}")
    private String elasticAddress;

    @Value("${csg_bill_es_cluster}")
    private String elasticCluster;


    private TransportClient transportClient;


    @PostConstruct
    public void init(){
        checkArgument(!Strings.isEmpty(elasticCluster) ,
         "ElasticSearch Cluster Name Null or EmptyString");
        TransportAddress[] transportAddresses = null;
        List<String> addressList = 
            Lists.newArrayList(
                Splitter.on(",").trimResults().omitEmptyStrings().split(elasticAddress)
        );
        checkArgument(!CollectionUtils.isEmpty(addressList) ,
         "ElasticSearch Cluster Address Can't be Empty");
        transportAddresses = new TransportAddress[addressList.size()];
        Splitter splitter = Splitter.on(":").omitEmptyStrings().trimResults();
        for(int i=0; i<addressList.size() ; i++){
            List<String> singleAddressPair = 
                Lists.newArrayList(splitter.split(addressList.get(i)));
            checkArgument(singleAddressPair.size()==2 ,
                    "ElasticSearch Address format address:port error" + addressList.get(i)
            );
            transportAddresses[i] = new InetSocketTransportAddress(
                    singleAddressPair.get(0),
                    Integer.parseInt(singleAddressPair.get(1))
            );
        }
        Settings settings = ImmutableSettings.builder()
            .put("cluster.name" , elasticCluster)
            .build();
        transportClient = new TransportClient(settings);
        transportClient.addTransportAddresses(transportAddresses);

        LOGGER.info("ElasticSearch Init Done ElasticAddress={} , ElasticClusterName={}" 
            , elasticAddress , elasticCluster);
    }

    @PreDestroy
    public void destroy() {
        if (transportClient != null) {
            transportClient.close();
        }
    }

    public TransportClient getTransportClient() {
        return transportClient;
    }

}

Service Method Interface

/**
 * Created by xueping.you on 15-7-30.
 */
@Service
public class ESBillDetailServiceBase {
    private final static Logger LOGGER = LoggerFactory.getLogger(ESBillDetailServiceBase.class);

    @Resource
    private ESTransportSessionFactory esTransportSessionFactory;

    @Value("${csg_bill_index_name}")
    private String ES_INDEX;

    @Value("${csg_bill_type_name}")
    private String ES_TYPE;

    private static ObjectMapper objectMapper = new ObjectMapper();

    static {
        objectMapper.setDateFormat(new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"));
    }

    /**
     * Method1 將訂單號 , 對賬單號 , 產品Id , 賬單號 ,結算供應商ID,結算對象Id,create_ts進行索引
     * @param esBillDetails
     */
    public void insertAllOnDuplicateIdCover(List<ESBillDetail> esBillDetails){
        Stopwatch stopwatch = Stopwatch.createStarted();
        if(CollectionUtils.isEmpty(esBillDetails)){
            return;
        }

        TransportClient client = esTransportSessionFactory.getTransportClient();
        BulkRequestBuilder bulkRequestBuilder = client.prepareBulk();

        for(ESBillDetail esBillDetail : esBillDetails){
            String source;
            try {
                source = objectMapper.writeValueAsString(esBillDetail);
            } catch (Exception e1) {
                LOGGER.error("create billDetail index error source : " + esBillDetail, e1);
                QMonitor.recordOne(QMonitorConstants.ES_BILL_DETAIL_INDEXING_ERROR);
                continue;
            }
            IndexRequestBuilder indexRequestBuilder = 
                client.prepareIndex(ES_INDEX, ES_TYPE, 
                esBillDetail.getId()).setSource(source);
            bulkRequestBuilder.add(indexRequestBuilder);
        }

        BulkResponse bulkResponse = bulkRequestBuilder.execute().actionGet();
        if(bulkResponse.hasFailures()){
            Iterator<BulkItemResponse> itemResponseIterator = bulkResponse.iterator();
            while(itemResponseIterator.hasNext()){
                BulkItemResponse itemResponse = itemResponseIterator.next();
                if(itemResponse!=null && itemResponse.isFailed()){
                    LOGGER.info("indexing billDetail error indexMessage={},
                        errorMassage={}",Joiner.on(',')                     .join(itemResponse.getIndex(),itemResponse.getType(),itemResponse.getId()),
                            JsonUtils.toJSONString(itemResponse));
                }
            }
            QMonitor.recordOne(QMonitorConstants.ES_BILL_DETAIL_INDEXING_ERROR);
        }

        QMonitor.recordOne(QMonitorConstants.ES_BILL_DETAIL_INDEXING_DONE , stopwatch.elapsed(TimeUnit.MILLISECONDS));
    }

    /**
     * Method2 通過訂單號 ，對賬單號, 產品Id , 賬單號 的任意條件進行查詢，支持分頁
     */
    public List<ESBillDetail> query(ESBillDetailQuery esBillDetailQuery , Pagination pagination){
        try {
            Stopwatch stopwatch = Stopwatch.createStarted();
            List<ESBillDetail> esBillDetails = Lists.newArrayList();
            BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
            for(Map.Entry entry : esBillDetailQuery.entrySet()){
                ESBillDetailQuery.Param param = (ESBillDetailQuery.Param)entry.getKey();
                param.decorateBoolQueryBuilder(entry.getValue(),boolQueryBuilder);
            }
            SearchRequestBuilder requestBuilder = esTransportSessionFactory.getTransportClient()
                    .prepareSearch(ES_INDEX)
                    .setTypes(ES_TYPE)
                    .setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
                    .setQuery(boolQueryBuilder);
            if(pagination!=null){
                    requestBuilder.setFrom(pagination.getStartIndex())
                            .setSize(pagination.getPageSize());
            }
            SearchResponse searchResponse = requestBuilder.execute().actionGet();
            SearchHits searchHits = searchResponse.getHits();
            for(SearchHit searchHit : searchHits){
                Map<String , Object> map = searchHit.getSource();
                ESBillDetail esBillDetail = objectMapper.readValue(
                        JsonUtils.toJSONString(map) ,
                        new TypeReference<ESBillDetail>() {});
                esBillDetails.add(esBillDetail);
            }
            LOGGER.info("Query ESDetail done query={} , time={}" , esBillDetailQuery , stopwatch.elapsed(TimeUnit.MILLISECONDS));
            QMonitor.recordOne(QMonitorConstants.ES_BILL_QUERY , stopwatch.elapsed(TimeUnit.MILLISECONDS));
            return esBillDetails;
        }catch (Exception e){
            LOGGER.error("elastic search query Error query={} , page={}" , esBillDetailQuery , pagination , e);
            QMonitor.recordOne(QMonitorConstants.ES_BILL_DETAIL_QUERY_ERROR);
            throw new RuntimeException("ES 查詢異常");
        }
    }

    /**
     * 此方法慎用！！！！！！！！！！！！！！！！！！！
     * Method3 通過匹配條件刪除ES 中的記錄
     */
    public boolean delete(ESBillDetailQuery esBillDetailQuery){
        try {
            Stopwatch stopwatch = Stopwatch.createStarted();
            BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
            for(Map.Entry entry : esBillDetailQuery.entrySet()){
                ESBillDetailQuery.Param param = (ESBillDetailQuery.Param)entry.getKey();
                param.decorateBoolQueryBuilder(entry.getValue(),boolQueryBuilder);
            }
            DeleteByQueryResponse deleteByQueryResponse = esTransportSessionFactory.getTransportClient()
                    .prepareDeleteByQuery(ES_INDEX )
                    .setTypes(ES_TYPE)
                    .setQuery(boolQueryBuilder).execute().actionGet();
            LOGGER.info("Delete ESDetail done query={} , time={}" , esBillDetailQuery , stopwatch.elapsed(TimeUnit.MILLISECONDS));
            QMonitor.recordOne(QMonitorConstants.ES_BILL_DELETE , stopwatch.elapsed(TimeUnit.MILLISECONDS));
            return deleteByQueryResponse.status().equals(RestStatus.OK) ? true : false;
        }catch (Exception e){
            LOGGER.error("elastic search delete Error query={} , page={}" , esBillDetailQuery , e);
            QMonitor.recordOne(QMonitorConstants.ES_BILL_DETAIL_QUERY_ERROR);
            throw new RuntimeException("ES 刪除記錄異常");
        }
    }

ESBillDetailQuery

/**
 * Created by xueping.you on 15-7-30.
 */
public class ESBillDetailQuery extends GenericQuery<ESBillDetailQuery.Param> {

    //全爲Str類型
    public enum Param{
        ids {
            @Override
            public <T, Q extends QueryBuilder> void decorateBoolQueryBuilder(T param, Q queryBuilder) {
                List<String> strsParam = (List<String>)param;
                BoolQueryBuilder boolQueryBuilder = (BoolQueryBuilder)queryBuilder;
                if(CollectionUtils.isEmpty(strsParam)){
                    return;
                }
                BoolQueryBuilder subCodeQuery = QueryBuilders.boolQuery();
                for(String strParam : strsParam){
                    subCodeQuery.should(QueryBuilders.matchQuery("billNo" , strParam).operator(MatchQueryBuilder.Operator.AND));
                }
                boolQueryBuilder.must(subCodeQuery);
            }
        },
        billNos {
            @Override
            public <T, Q extends QueryBuilder> void decorateBoolQueryBuilder(T param, Q queryBuilder) {
                List<String> strsParam = (List<String>)param;
                BoolQueryBuilder boolQueryBuilder = (BoolQueryBuilder)queryBuilder;
                if(CollectionUtils.isEmpty(strsParam)){
                    return;
                }
                BoolQueryBuilder subCodeQuery = QueryBuilders.boolQuery();
                for(String strParam : strsParam){
                    subCodeQuery.should(QueryBuilders.matchQuery("billNo" , strParam).operator(MatchQueryBuilder.Operator.AND));
                }
                boolQueryBuilder.must(subCodeQuery);
            }
        },
        orderNos {
            @Override
            public <T, Q extends QueryBuilder> void decorateBoolQueryBuilder(T param, Q queryBuilder) {
                List<String> strsParam = (List<String>)param;
                BoolQueryBuilder boolQueryBuilder = (BoolQueryBuilder)queryBuilder;
                if(CollectionUtils.isEmpty(strsParam)){
                    return;
                }
                BoolQueryBuilder subCodeQuery = QueryBuilders.boolQuery();
                for(String strParam : strsParam){
                    subCodeQuery.should(QueryBuilders.matchQuery("orderNo" , strParam).operator(MatchQueryBuilder.Operator.AND));
                }
                boolQueryBuilder.must(subCodeQuery);
            }
        },
        productIds {
            @Override
            public <T, Q extends QueryBuilder> void decorateBoolQueryBuilder(T param, Q queryBuilder) {
                List<String> strsParam = (List<String>)param;
                BoolQueryBuilder boolQueryBuilder = (BoolQueryBuilder)queryBuilder;
                if(CollectionUtils.isEmpty(strsParam)){
                    return;
                }
                BoolQueryBuilder subCodeQuery = QueryBuilders.boolQuery();
                for(String strParam : strsParam){
                    subCodeQuery.should(QueryBuilders.matchQuery("productId" , strParam).operator(MatchQueryBuilder.Operator.AND));
                }
                boolQueryBuilder.must(subCodeQuery);
            }
        },
        reconcileIds {
            @Override
            public <T, Q extends QueryBuilder> void decorateBoolQueryBuilder(T param, Q queryBuilder) {
                List<String> strsParam = (List<String>)param;
                BoolQueryBuilder boolQueryBuilder = (BoolQueryBuilder)queryBuilder;
                if(CollectionUtils.isEmpty(strsParam)){
                    return;
                }
                BoolQueryBuilder subCodeQuery = QueryBuilders.boolQuery();
                for(String strParam : strsParam){
                    subCodeQuery.should(QueryBuilders.matchQuery("reconcileId" , strParam).operator(MatchQueryBuilder.Operator.AND));
                }
                boolQueryBuilder.must(subCodeQuery);
            }
        };

        public abstract <T, Q extends QueryBuilder> void decorateBoolQueryBuilder(T param ,  Q queryBuilder );
    }


    public ESBillDetailQuery() {
    }

    public ESBillDetailQuery(Integer startIndex, Integer maxCount) {
        super(startIndex, maxCount);
    }

    public ESBillDetailQuery(Class<Param> paramType) {
        super(paramType);
    }
}

ElasticSearch簡單使用

What is ElasticSearch ？

Config elasticSearch.properties

Create elasticSearch TransportSessionFactory

Service Method Interface

ESBillDetailQuery

《Python進階》學習筆記

一個docker容器暴露多個端口

leetcode 60 排列序列

微服務實踐之使用 Visual Studio 2022 調試Dapr 應用程序

wpf附加屬性理解 WPF附加屬性

Java NIO 特性學習

Shell查詢數據庫，和發送郵件

MySQL Cluster(MySQL 集羣) 初試

Linux 基礎命令知識1

Guava EventBus配合Spring完成事件分發可配置(變通使用三方庫)

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結