Spring Cloud | ElasticSearch（es）的使用

一、簡述

ES(ElasticSearch)是一款分佈式全文檢索框架，每個字段可以被索引與搜索；以勝任上百個服務節點的擴展，並支持PB級別的結構化或者非結構化數據；底層基於基於Lucene實現。

ES與傳統數據的區別：

1、結構名稱不同：一個ES集羣可以包含多個索引（數據庫），每個索引又包含了很多類型（表），類型中包含了很多文檔（行），每個文檔使用 JSON 格式存儲數據，包含了很多字段（列）。

關係型數據庫	數據庫	表	行	列
ElasticSearch	索引	類型	文檔	字段

2、ES分佈式搜索，傳統數據庫遍歷式搜索

3、ES採用倒排索引，傳統數據庫採用B+樹索引

4、ES沒有用戶驗證和權限控制

5、ES沒有事務的概念，不支持回滾，誤刪不能恢復

......

本章案例源碼：

源碼：https://github.com/liujun19921020/SpringCloudDemo/blob/master/ProjectDemo/企業SpringCloud架構-xxljob-redis-elasticsearch

或 ：鏈接：https://pan.baidu.com/s/1ooqaIRSeX6naOZ51aBWNHA  提取碼：cwsw

 

這裏是整了一個架構集合案例，該文主要源碼爲sc-elasticsearch-demo項目。

 

二、目標

通過接口傳遞list對象，將集合數據插入ES索引名/類型名爲”order_detail”對象中；然後再通過接口按條件查詢。

（下面講述通過插入數據時自動創建ES索引及mapping方式，也可通過sql+conf文件方式獲取數據源創建ES相關，詳見源碼目錄）

 

三、配置

項目中我們採用SearchGurageSSL認證方式配置，證書配置生成的可查看《SearchGuard證書配置》、《searchguard配置》、《SearchGuard客戶端連接》、《SearchGuard集羣版》等相關文檔。

 

先引入ES配置

1、pom.xml中引入依賴

<!-- es插件 -->
<dependency>
    <groupId>org.springframework.data</groupId>
    <artifactId>spring-data-elasticsearch</artifactId>
    <version>3.0.4.RELEASE</version>
</dependency>
<dependency>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch</artifactId>
    <version>5.6.14</version>
</dependency>

<!--SearchGurageSSL認證-->
<dependency>
    <groupId>com.floragunn</groupId>
    <artifactId>search-guard-ssl</artifactId>
    <version>5.6.14-23</version>
</dependency>

2、在application.properties配置文件中添加配置變量，指明ssl證書存放位置、ES的IP端口及賬號密碼

#----------------------------------elasticsearch配置---------------------------------------------
# resources.base_location=D:\\elasticsearch\\
resources.base_location=classpath:
resources.locations1=${resources.base_location}ssl/spock.key
resources.locations2=${resources.base_location}ssl/spock.pem
resources.locations3=${resources.base_location}ssl/root-ca.pem

elasticsearch.number=1
elasticsearch.host1=192.168.71.246
elasticsearch.host2=192.168.71.246
elasticsearch.host3=192.168.71.246
elasticsearch.port=9300

elasticsearch.cluster-name=ebuy-cloud-cluster
elasticsearch.ssl.transport.pemkey.password=3QgfFoYd8Ken

 

3、編寫ES的ElasticConfig.java文件，通過上述的配置資料，創建TransportClient，初始化Bean中（@EnableElasticsearchRepositories註解後配置的包路徑即掃描可使用ES對象地址）

/**
 * es配置
 */
@Configuration
@EnableElasticsearchRepositories(basePackages = "com.lj.scelasticsearchdemo.service")
public class ElasticConfig {
    @Value("${elasticsearch.number}")
    private Integer number;
    @Value("${elasticsearch.host1}")
    private String host1;
    @Value("${elasticsearch.host2}")
    private String host2;
    @Value("${elasticsearch.host3}")
    private String host3;
    @Value("${elasticsearch.port}")
    private Integer port;
    @Value("${elasticsearch.cluster-name}")
    private String clusterName;
    @Value("${elasticsearch.ssl.transport.pemkey.password}")
    private String password;
    @Value("${resources.locations1}")
    private String resourcesLocations1;
    @Value("${resources.locations2}")
    private String resourcesLocations2;
    @Value("${resources.locations3}")
    private String resourcesLocations3;
    /**
     * 注入的ElasticSearch實例
     */
    @Bean(name = "esClient")
    public TransportClient getclient()throws Exception {

        Settings settings = Settings.builder()
                // 本地開發使用
                .put(SSLConfigConstants.SEARCHGUARD_SSL_TRANSPORT_PEMKEY_FILEPATH, ResourceUtils.getFile(resourcesLocations1))
                .put(SSLConfigConstants.SEARCHGUARD_SSL_TRANSPORT_PEMCERT_FILEPATH, ResourceUtils.getFile(resourcesLocations2))
                .put(SSLConfigConstants.SEARCHGUARD_SSL_TRANSPORT_PEMTRUSTEDCAS_FILEPATH, ResourceUtils.getFile(resourcesLocations3))
                .put(SSLConfigConstants.SEARCHGUARD_SSL_TRANSPORT_PEMKEY_PASSWORD, password)
                .put("cluster.name",clusterName)
                .build();
        TransportClient client = null;
        if(1==number){
            client = new PreBuiltTransportClient(settings, SearchGuardSSLPlugin.class)
                    .addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(host1), port));
        }else if(3==number){
            client = new PreBuiltTransportClient(settings, SearchGuardSSLPlugin.class)
                    .addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(host1), port))
                    .addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(host2), port))
                    .addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(host3), port));
        }else{
            System.out.println("輸入節點數量異常！");
            return client;
        }
        // 獲取連接
//        client.admin().cluster().nodesInfo(new NodesInfoRequest()).actionGet();

        return client;
    }
}

4、ES主要配置基本上已完成，我們可以寫個繼承AbstractResultMapper的類，作爲查詢ES後的自定義結果映射類

/**
 * 類名稱：ExtResultMapper
 * 類描述：自定義結果映射類
 */
@Component
public class ExtResultMapper extends AbstractResultMapper {

    private static final Logger logger = LoggerFactory.getLogger(ExtResultMapper.class);

    private MappingContext<? extends ElasticsearchPersistentEntity<?>, ElasticsearchPersistentProperty> mappingContext;

    public ExtResultMapper() {
        super(new DefaultEntityMapper());
    }

    public ExtResultMapper(MappingContext<? extends ElasticsearchPersistentEntity<?>, ElasticsearchPersistentProperty> mappingContext) {
        super(new DefaultEntityMapper());
        this.mappingContext = mappingContext;
    }

    public ExtResultMapper(EntityMapper entityMapper) {
        super(entityMapper);
    }

    public ExtResultMapper(
            MappingContext<? extends ElasticsearchPersistentEntity<?>, ElasticsearchPersistentProperty> mappingContext,
            EntityMapper entityMapper) {
        super(entityMapper);
        this.mappingContext = mappingContext;
    }

    @Override
    public <T> AggregatedPage<T> mapResults(SearchResponse response, Class<T> clazz, Pageable pageable) {
        long totalHits = response.getHits().getTotalHits();
        List<T> results = new ArrayList<>();
        for (SearchHit hit : response.getHits()) {
            if (hit != null) {
                T result = null;
                if (StringUtils.hasText(hit.getSourceAsString())) {
                    result = JSONObject.parseObject(hit.getSourceAsString(),clazz);
//                    result = mapEntity(hit.sourceAsString(), clazz);
                } else {
                    result = JSONObject.parseObject(hit.getSourceAsString(),clazz);
//                    result = mapEntity(hit.getFields().values(), clazz);
                }
                setPersistentEntityId(result, hit.getId(), clazz);
                setPersistentEntityVersion(result, hit.getVersion(), clazz);
                populateScriptFields(result, hit);
                if (!org.apache.commons.lang.StringUtils.isBlank(hit.getId())){
                    //設置文檔id
                    JSONObject jsonObject = JSONObject.parseObject(JSONObject.toJSONString(result));
                    jsonObject.put("esId",hit.getId());
                    result = jsonObject.toJavaObject(clazz);
                }


                // 高亮查詢
                populateHighLightedFields(result, hit.getHighlightFields());
                results.add(result);
            }
        }

        return new AggregatedPageImpl<T>(results, pageable, totalHits, response.getAggregations(), response.getScrollId());
    }

    private <T>  void populateHighLightedFields(T result, Map<String, HighlightField> highlightFields) {
        for (HighlightField field : highlightFields.values()) {
            try {
                PropertyUtils.setProperty(result, field.getName(), concat(field.fragments()));
            } catch (InvocationTargetException | IllegalAccessException | NoSuchMethodException e) {
                throw new ElasticsearchException("failed to set highlighted value for field: " + field.getName()
                        + " with value: " + Arrays.toString(field.getFragments()), e);
            }
        }
    }

    private String concat(Text[] texts) {
        StringBuffer sb = new StringBuffer();
        for (Text text : texts) {
            sb.append(text.toString());
        }
        return sb.toString();
    }

    private <T> void populateScriptFields(T result, SearchHit hit) {
        if (hit.getFields() != null && !hit.getFields().isEmpty() && result != null) {
            for (Field field : result.getClass().getDeclaredFields()) {
                ScriptedField scriptedField = field.getAnnotation(ScriptedField.class);
                if (scriptedField != null) {
                    String name = scriptedField.name().isEmpty() ? field.getName() : scriptedField.name();
                    SearchHitField searchHitField = hit.getFields().get(name);
                    if (searchHitField != null) {
                        field.setAccessible(true);
                        try {
                            field.set(result, searchHitField.getValue());
                        } catch (IllegalArgumentException e) {
                            throw new ElasticsearchException("failed to set scripted field: " + name + " with value: "
                                    + searchHitField.getValue(), e);
                        } catch (IllegalAccessException e) {
                            throw new ElasticsearchException("failed to access scripted field: " + name, e);
                        }
                    }
                }
            }
        }
    }

    private <T> T mapEntity(Collection<SearchHitField> values, Class<T> clazz) {
        return mapEntity(buildJSONFromFields(values), clazz);
    }

    private String buildJSONFromFields(Collection<SearchHitField> values) {
        JsonFactory nodeFactory = new JsonFactory();
        try {
            ByteArrayOutputStream stream = new ByteArrayOutputStream();
            JsonGenerator generator = nodeFactory.createGenerator(stream, JsonEncoding.UTF8);
            generator.writeStartObject();
            for (SearchHitField value : values) {
                if (value.getValues().size() > 1) {
                    generator.writeArrayFieldStart(value.getName());
                    for (Object val : value.getValues()) {
                        generator.writeObject(val);
                    }
                    generator.writeEndArray();
                } else {
                    generator.writeObjectField(value.getName(), value.getValue());
                }
            }
            generator.writeEndObject();
            generator.flush();
            return new String(stream.toByteArray(), Charset.forName("UTF-8"));
        } catch (IOException e) {
            return null;
        }
    }

    @Override
    public <T> T mapResult(GetResponse response, Class<T> clazz) {
        T result = mapEntity(response.getSourceAsString(), clazz);
        if (result != null) {
            setPersistentEntityId(result, response.getId(), clazz);
            setPersistentEntityVersion(result, response.getVersion(), clazz);
        }
        return result;
    }

    @Override
    public <T> LinkedList<T> mapResults(MultiGetResponse responses, Class<T> clazz) {
        LinkedList<T> list = new LinkedList<>();
        for (MultiGetItemResponse response : responses.getResponses()) {
            if (!response.isFailed() && response.getResponse().isExists()) {
                T result = mapEntity(response.getResponse().getSourceAsString(), clazz);
                setPersistentEntityId(result, response.getResponse().getId(), clazz);
                setPersistentEntityVersion(result, response.getResponse().getVersion(), clazz);
                list.add(result);
            }
        }
        return list;
    }

    private <T> void setPersistentEntityId(T result, String id, Class<T> clazz) {

        if (mappingContext != null && clazz.isAnnotationPresent(Document.class)) {
            ElasticsearchPersistentEntity<?> persistentEntity = mappingContext.getRequiredPersistentEntity(clazz);
            ElasticsearchPersistentProperty idProperty = persistentEntity.getIdProperty();

            // Only deal with String because ES generated Ids are strings !
            if (idProperty != null && idProperty.getType().isAssignableFrom(String.class)) {
                persistentEntity.getPropertyAccessor(result).setProperty(idProperty, id);
            }
        }
    }

    private <T> void setPersistentEntityVersion(T result, long version, Class<T> clazz) {
        if (mappingContext != null && clazz.isAnnotationPresent(Document.class)) {

            ElasticsearchPersistentEntity<?> persistentEntity = mappingContext.getPersistentEntity(clazz);
            ElasticsearchPersistentProperty versionProperty = persistentEntity.getVersionProperty();

            // Only deal with Long because ES versions are longs !
            if (versionProperty != null && versionProperty.getType().isAssignableFrom(Long.class)) {
                // check that a version was actually returned in the response, -1 would indicate that
                // a search didn't request the version ids in the response, which would be an issue
                Assert.isTrue(version != -1, "Version in response is -1");
                persistentEntity.getPropertyAccessor(result).setProperty(versionProperty, version);
            }
        }
    }
}

 

四、業務代碼

1、service實現插入\查詢，置於之前配置的掃描路徑下（查詢的類似sql語句爲見註釋說明）。

@Service
public class EsDemoServiceImpl implements EsDemoService {

    private final static String ES_INDEX = "order_detail";//ES索引名
    private final static String ES_TYPE = "order_detail";//ES類型名

    @Autowired
    private ElasticsearchTemplate elasticsearchTemplate;

    @Autowired
    private TransportClient esClient;

    @Resource
    private ExtResultMapper extResultMapper;

    /**
     * 添加es案例信息 （數據來源爲多庫時 可以通到java代碼實現數據插入；數據來源爲同一機器時，建議通過sql+conf的形式導入數據）
     * @author Liujun
     * @param pushMapList
     * @return
     */
    @Override
    public Integer saveEsDemoInfo(List<Map<String, Object>> pushMapList) {
        Integer saveNum = 0;//返回的插入多少條數據
        for (Map<String, Object> pushMap : pushMapList) {
            pushMap.put("id",System.currentTimeMillis()+new Random().nextInt(100));//id不能相等，因爲要多次插入一條數據,我們生成不一樣的id
            IndexResponse response = esClient.prepareIndex(ES_INDEX, ES_TYPE)
                    .setSource(pushMap)
                    .get();
            String _id = response.getId();
            if (StringUtils.isNotBlank(_id)){
                saveNum++;
            }
        }
        return saveNum;
    }


    /**
     * 查詢es案例信息 （假設條件爲類似sql:select order_id from sku=xx and site=xx and (order_status=1 or order_status = 2)）
     * @param pageable
     * @param sku
     * @param site
     * @return
     */
    @Override
    public PageMap selectEsDemoInfo(Pageable pageable, String sku, String site) {
        //創建builder
        BoolQueryBuilder builder = QueryBuilders.boolQuery();
        //builder下有must、should以及mustNot 相當於sql中的and、or以及not

        //設置模糊搜索
        if (!StringUtils.isBlank(sku)) {
            builder.must(QueryBuilders.termQuery("sku.keyword", sku));
        }
        if (!StringUtils.isBlank(site)) {
            builder.must(QueryBuilders.termQuery("site.keyword", site));
        }

        BoolQueryBuilder purBuilder = QueryBuilders.boolQuery();
        purBuilder.should(QueryBuilders.matchPhraseQuery("order_status", "1"));
        purBuilder.should(QueryBuilders.matchPhraseQuery("order_status", "2"));
        builder.must(purBuilder);

        //構建查詢
        NativeSearchQueryBuilder nativeSearchQueryBuilder = new NativeSearchQueryBuilder();
        //將搜索條件設置到構建中
        nativeSearchQueryBuilder.withQuery(builder);
        //將分頁設置到構建中
        nativeSearchQueryBuilder.withPageable(pageable);

        //生產NativeSearchQuery
        NativeSearchQuery query = nativeSearchQueryBuilder.build();
        //執行,返回結果的分頁
        Page<EsOrderDetail> resutlList = elasticsearchTemplate.queryForPage(query, EsOrderDetail.class, extResultMapper);
        return EsPageTool.getPageMap(resutlList, pageable);
    }

}

 

2、上面查詢的映射類，在實體類上表名ES索引、類型

/**
 * 查詢的es分片對應的index/type的字段
 */
@Document(indexName = "order_detail", type = "order_detail")
public class EsOrderDetail implements Serializable {
    private String order_id;
    private String order_status;
}

3、貼一下Controller接口

/**
 * ES 案例
 */
@RestController
@RequestMapping("/esDemo")
public class EsDemoController {

    private static Logger logger = LoggerFactory.getLogger(EsDemoController.class);

    @Autowired
    private EsDemoService esDemoService;

    /**
     * 查詢es案例信息 （假設條件爲類似sql:select order_id from sku=xx and site=xx and (order_status=1 or order_status = 2)）
     * @param jsonObject
     * @author Liujun
     * @return
     */
    @PostMapping("/selectEsDemoInfo")
    public ResponseMsg selectEsDemoInfo(@RequestBody JSONObject jsonObject) {
        try {
            Integer page = jsonObject.getInteger("page");
            Integer size = jsonObject.getInteger("size");
            String sku = jsonObject.getString("sku");
            String site = jsonObject.getString("site");

            Pageable pageable = PageRequest.of(page == null ? 0 : page-1, size == null ? 50 : size);//分頁機制
            PageMap pageMap = esDemoService.selectEsDemoInfo(pageable, sku, site);
            return new ResponseMsg(Code.SUCCESS, pageMap, "根據條件查詢ES案例信息成功！");
        } catch (Exception e) {
            logger.error("EsDemoController.selectEsDemoInfo 異常", e);
            return new ResponseMsg(Code.FAIL, null, "根據條件查詢ES案例信息失敗！"+e.getMessage());
        }
    }

    /**
     * 添加es案例信息 （數據來源爲多庫時 可以通到java代碼實現數據插入；數據來源爲同一機器時，建議通過sql+conf的形式導入數據）
     * @param pushMapList
     * @author Liujun
     * @return
     */
    @PostMapping("/saveEsDemoInfo")
    public ResponseMsg saveEsDemoInfo(@RequestBody List<Map<String, Object>> pushMapList){
        try{
            Integer saveNum = esDemoService.saveEsDemoInfo(pushMapList);
            return new ResponseMsg(Code.SUCCESS, saveNum,"添加ES案例信息成功！");
        } catch (Exception e){
            logger.error("EsDemoController.saveEsDemoInfo 異常", e);
            return new ResponseMsg(Code.FAIL,null, "添加ES案例信息失敗！"+e.getMessage());
        }
    }
}

 

4、依次來看看我們的結果（這裏我們通過Kibana來操作ES）

執行之前，發現ES索引是不存在的

 

調用接口，插入list集合

 

此時再查看ES索引，發現已經生成了數據

 

調用查詢接口，查詢到分頁數據（ES查詢必帶分頁，若不寫分頁代碼，ES也會默認分頁10條數據）