Spring Cloud | ElasticSearch（es）的使用

一、简述

ES(ElasticSearch)是一款分布式全文检索框架，每个字段可以被索引与搜索；以胜任上百个服务节点的扩展，并支持PB级别的结构化或者非结构化数据；底层基于基于Lucene实现。

ES与传统数据的区别：

1、结构名称不同：一个ES集群可以包含多个索引（数据库），每个索引又包含了很多类型（表），类型中包含了很多文档（行），每个文档使用 JSON 格式存储数据，包含了很多字段（列）。

关系型数据库	数据库	表	行	列
ElasticSearch	索引	类型	文档	字段

2、ES分布式搜索，传统数据库遍历式搜索

3、ES采用倒排索引，传统数据库采用B+树索引

4、ES没有用户验证和权限控制

5、ES没有事务的概念，不支持回滚，误删不能恢复

......

本章案例源码：

源码：https://github.com/liujun19921020/SpringCloudDemo/blob/master/ProjectDemo/企业SpringCloud架构-xxljob-redis-elasticsearch

或 ：链接：https://pan.baidu.com/s/1ooqaIRSeX6naOZ51aBWNHA  提取码：cwsw

 

这里是整了一个架构集合案例，该文主要源码为sc-elasticsearch-demo项目。

 

二、目标

通过接口传递list对象，将集合数据插入ES索引名/类型名为”order_detail”对象中；然后再通过接口按条件查询。

（下面讲述通过插入数据时自动创建ES索引及mapping方式，也可通过sql+conf文件方式获取数据源创建ES相关，详见源码目录）

 

三、配置

项目中我们采用SearchGurageSSL认证方式配置，证书配置生成的可查看《SearchGuard证书配置》、《searchguard配置》、《SearchGuard客户端连接》、《SearchGuard集群版》等相关文档。

 

先引入ES配置

1、pom.xml中引入依赖

<!-- es插件 -->
<dependency>
    <groupId>org.springframework.data</groupId>
    <artifactId>spring-data-elasticsearch</artifactId>
    <version>3.0.4.RELEASE</version>
</dependency>
<dependency>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch</artifactId>
    <version>5.6.14</version>
</dependency>

<!--SearchGurageSSL认证-->
<dependency>
    <groupId>com.floragunn</groupId>
    <artifactId>search-guard-ssl</artifactId>
    <version>5.6.14-23</version>
</dependency>

2、在application.properties配置文件中添加配置变量，指明ssl证书存放位置、ES的IP端口及账号密码

#----------------------------------elasticsearch配置---------------------------------------------
# resources.base_location=D:\\elasticsearch\\
resources.base_location=classpath:
resources.locations1=${resources.base_location}ssl/spock.key
resources.locations2=${resources.base_location}ssl/spock.pem
resources.locations3=${resources.base_location}ssl/root-ca.pem

elasticsearch.number=1
elasticsearch.host1=192.168.71.246
elasticsearch.host2=192.168.71.246
elasticsearch.host3=192.168.71.246
elasticsearch.port=9300

elasticsearch.cluster-name=ebuy-cloud-cluster
elasticsearch.ssl.transport.pemkey.password=3QgfFoYd8Ken

 

3、编写ES的ElasticConfig.java文件，通过上述的配置资料，创建TransportClient，初始化Bean中（@EnableElasticsearchRepositories注解后配置的包路径即扫描可使用ES对象地址）

/**
 * es配置
 */
@Configuration
@EnableElasticsearchRepositories(basePackages = "com.lj.scelasticsearchdemo.service")
public class ElasticConfig {
    @Value("${elasticsearch.number}")
    private Integer number;
    @Value("${elasticsearch.host1}")
    private String host1;
    @Value("${elasticsearch.host2}")
    private String host2;
    @Value("${elasticsearch.host3}")
    private String host3;
    @Value("${elasticsearch.port}")
    private Integer port;
    @Value("${elasticsearch.cluster-name}")
    private String clusterName;
    @Value("${elasticsearch.ssl.transport.pemkey.password}")
    private String password;
    @Value("${resources.locations1}")
    private String resourcesLocations1;
    @Value("${resources.locations2}")
    private String resourcesLocations2;
    @Value("${resources.locations3}")
    private String resourcesLocations3;
    /**
     * 注入的ElasticSearch实例
     */
    @Bean(name = "esClient")
    public TransportClient getclient()throws Exception {

        Settings settings = Settings.builder()
                // 本地开发使用
                .put(SSLConfigConstants.SEARCHGUARD_SSL_TRANSPORT_PEMKEY_FILEPATH, ResourceUtils.getFile(resourcesLocations1))
                .put(SSLConfigConstants.SEARCHGUARD_SSL_TRANSPORT_PEMCERT_FILEPATH, ResourceUtils.getFile(resourcesLocations2))
                .put(SSLConfigConstants.SEARCHGUARD_SSL_TRANSPORT_PEMTRUSTEDCAS_FILEPATH, ResourceUtils.getFile(resourcesLocations3))
                .put(SSLConfigConstants.SEARCHGUARD_SSL_TRANSPORT_PEMKEY_PASSWORD, password)
                .put("cluster.name",clusterName)
                .build();
        TransportClient client = null;
        if(1==number){
            client = new PreBuiltTransportClient(settings, SearchGuardSSLPlugin.class)
                    .addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(host1), port));
        }else if(3==number){
            client = new PreBuiltTransportClient(settings, SearchGuardSSLPlugin.class)
                    .addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(host1), port))
                    .addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(host2), port))
                    .addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(host3), port));
        }else{
            System.out.println("输入节点数量异常！");
            return client;
        }
        // 获取连接
//        client.admin().cluster().nodesInfo(new NodesInfoRequest()).actionGet();

        return client;
    }
}

4、ES主要配置基本上已完成，我们可以写个继承AbstractResultMapper的类，作为查询ES后的自定义结果映射类

/**
 * 类名称：ExtResultMapper
 * 类描述：自定义结果映射类
 */
@Component
public class ExtResultMapper extends AbstractResultMapper {

    private static final Logger logger = LoggerFactory.getLogger(ExtResultMapper.class);

    private MappingContext<? extends ElasticsearchPersistentEntity<?>, ElasticsearchPersistentProperty> mappingContext;

    public ExtResultMapper() {
        super(new DefaultEntityMapper());
    }

    public ExtResultMapper(MappingContext<? extends ElasticsearchPersistentEntity<?>, ElasticsearchPersistentProperty> mappingContext) {
        super(new DefaultEntityMapper());
        this.mappingContext = mappingContext;
    }

    public ExtResultMapper(EntityMapper entityMapper) {
        super(entityMapper);
    }

    public ExtResultMapper(
            MappingContext<? extends ElasticsearchPersistentEntity<?>, ElasticsearchPersistentProperty> mappingContext,
            EntityMapper entityMapper) {
        super(entityMapper);
        this.mappingContext = mappingContext;
    }

    @Override
    public <T> AggregatedPage<T> mapResults(SearchResponse response, Class<T> clazz, Pageable pageable) {
        long totalHits = response.getHits().getTotalHits();
        List<T> results = new ArrayList<>();
        for (SearchHit hit : response.getHits()) {
            if (hit != null) {
                T result = null;
                if (StringUtils.hasText(hit.getSourceAsString())) {
                    result = JSONObject.parseObject(hit.getSourceAsString(),clazz);
//                    result = mapEntity(hit.sourceAsString(), clazz);
                } else {
                    result = JSONObject.parseObject(hit.getSourceAsString(),clazz);
//                    result = mapEntity(hit.getFields().values(), clazz);
                }
                setPersistentEntityId(result, hit.getId(), clazz);
                setPersistentEntityVersion(result, hit.getVersion(), clazz);
                populateScriptFields(result, hit);
                if (!org.apache.commons.lang.StringUtils.isBlank(hit.getId())){
                    //设置文档id
                    JSONObject jsonObject = JSONObject.parseObject(JSONObject.toJSONString(result));
                    jsonObject.put("esId",hit.getId());
                    result = jsonObject.toJavaObject(clazz);
                }


                // 高亮查询
                populateHighLightedFields(result, hit.getHighlightFields());
                results.add(result);
            }
        }

        return new AggregatedPageImpl<T>(results, pageable, totalHits, response.getAggregations(), response.getScrollId());
    }

    private <T>  void populateHighLightedFields(T result, Map<String, HighlightField> highlightFields) {
        for (HighlightField field : highlightFields.values()) {
            try {
                PropertyUtils.setProperty(result, field.getName(), concat(field.fragments()));
            } catch (InvocationTargetException | IllegalAccessException | NoSuchMethodException e) {
                throw new ElasticsearchException("failed to set highlighted value for field: " + field.getName()
                        + " with value: " + Arrays.toString(field.getFragments()), e);
            }
        }
    }

    private String concat(Text[] texts) {
        StringBuffer sb = new StringBuffer();
        for (Text text : texts) {
            sb.append(text.toString());
        }
        return sb.toString();
    }

    private <T> void populateScriptFields(T result, SearchHit hit) {
        if (hit.getFields() != null && !hit.getFields().isEmpty() && result != null) {
            for (Field field : result.getClass().getDeclaredFields()) {
                ScriptedField scriptedField = field.getAnnotation(ScriptedField.class);
                if (scriptedField != null) {
                    String name = scriptedField.name().isEmpty() ? field.getName() : scriptedField.name();
                    SearchHitField searchHitField = hit.getFields().get(name);
                    if (searchHitField != null) {
                        field.setAccessible(true);
                        try {
                            field.set(result, searchHitField.getValue());
                        } catch (IllegalArgumentException e) {
                            throw new ElasticsearchException("failed to set scripted field: " + name + " with value: "
                                    + searchHitField.getValue(), e);
                        } catch (IllegalAccessException e) {
                            throw new ElasticsearchException("failed to access scripted field: " + name, e);
                        }
                    }
                }
            }
        }
    }

    private <T> T mapEntity(Collection<SearchHitField> values, Class<T> clazz) {
        return mapEntity(buildJSONFromFields(values), clazz);
    }

    private String buildJSONFromFields(Collection<SearchHitField> values) {
        JsonFactory nodeFactory = new JsonFactory();
        try {
            ByteArrayOutputStream stream = new ByteArrayOutputStream();
            JsonGenerator generator = nodeFactory.createGenerator(stream, JsonEncoding.UTF8);
            generator.writeStartObject();
            for (SearchHitField value : values) {
                if (value.getValues().size() > 1) {
                    generator.writeArrayFieldStart(value.getName());
                    for (Object val : value.getValues()) {
                        generator.writeObject(val);
                    }
                    generator.writeEndArray();
                } else {
                    generator.writeObjectField(value.getName(), value.getValue());
                }
            }
            generator.writeEndObject();
            generator.flush();
            return new String(stream.toByteArray(), Charset.forName("UTF-8"));
        } catch (IOException e) {
            return null;
        }
    }

    @Override
    public <T> T mapResult(GetResponse response, Class<T> clazz) {
        T result = mapEntity(response.getSourceAsString(), clazz);
        if (result != null) {
            setPersistentEntityId(result, response.getId(), clazz);
            setPersistentEntityVersion(result, response.getVersion(), clazz);
        }
        return result;
    }

    @Override
    public <T> LinkedList<T> mapResults(MultiGetResponse responses, Class<T> clazz) {
        LinkedList<T> list = new LinkedList<>();
        for (MultiGetItemResponse response : responses.getResponses()) {
            if (!response.isFailed() && response.getResponse().isExists()) {
                T result = mapEntity(response.getResponse().getSourceAsString(), clazz);
                setPersistentEntityId(result, response.getResponse().getId(), clazz);
                setPersistentEntityVersion(result, response.getResponse().getVersion(), clazz);
                list.add(result);
            }
        }
        return list;
    }

    private <T> void setPersistentEntityId(T result, String id, Class<T> clazz) {

        if (mappingContext != null && clazz.isAnnotationPresent(Document.class)) {
            ElasticsearchPersistentEntity<?> persistentEntity = mappingContext.getRequiredPersistentEntity(clazz);
            ElasticsearchPersistentProperty idProperty = persistentEntity.getIdProperty();

            // Only deal with String because ES generated Ids are strings !
            if (idProperty != null && idProperty.getType().isAssignableFrom(String.class)) {
                persistentEntity.getPropertyAccessor(result).setProperty(idProperty, id);
            }
        }
    }

    private <T> void setPersistentEntityVersion(T result, long version, Class<T> clazz) {
        if (mappingContext != null && clazz.isAnnotationPresent(Document.class)) {

            ElasticsearchPersistentEntity<?> persistentEntity = mappingContext.getPersistentEntity(clazz);
            ElasticsearchPersistentProperty versionProperty = persistentEntity.getVersionProperty();

            // Only deal with Long because ES versions are longs !
            if (versionProperty != null && versionProperty.getType().isAssignableFrom(Long.class)) {
                // check that a version was actually returned in the response, -1 would indicate that
                // a search didn't request the version ids in the response, which would be an issue
                Assert.isTrue(version != -1, "Version in response is -1");
                persistentEntity.getPropertyAccessor(result).setProperty(versionProperty, version);
            }
        }
    }
}

 

四、业务代码

1、service实现插入\查询，置于之前配置的扫描路径下（查询的类似sql语句为见注释说明）。

@Service
public class EsDemoServiceImpl implements EsDemoService {

    private final static String ES_INDEX = "order_detail";//ES索引名
    private final static String ES_TYPE = "order_detail";//ES类型名

    @Autowired
    private ElasticsearchTemplate elasticsearchTemplate;

    @Autowired
    private TransportClient esClient;

    @Resource
    private ExtResultMapper extResultMapper;

    /**
     * 添加es案例信息 （数据来源为多库时 可以通到java代码实现数据插入；数据来源为同一机器时，建议通过sql+conf的形式导入数据）
     * @author Liujun
     * @param pushMapList
     * @return
     */
    @Override
    public Integer saveEsDemoInfo(List<Map<String, Object>> pushMapList) {
        Integer saveNum = 0;//返回的插入多少条数据
        for (Map<String, Object> pushMap : pushMapList) {
            pushMap.put("id",System.currentTimeMillis()+new Random().nextInt(100));//id不能相等，因为要多次插入一条数据,我们生成不一样的id
            IndexResponse response = esClient.prepareIndex(ES_INDEX, ES_TYPE)
                    .setSource(pushMap)
                    .get();
            String _id = response.getId();
            if (StringUtils.isNotBlank(_id)){
                saveNum++;
            }
        }
        return saveNum;
    }


    /**
     * 查询es案例信息 （假设条件为类似sql:select order_id from sku=xx and site=xx and (order_status=1 or order_status = 2)）
     * @param pageable
     * @param sku
     * @param site
     * @return
     */
    @Override
    public PageMap selectEsDemoInfo(Pageable pageable, String sku, String site) {
        //创建builder
        BoolQueryBuilder builder = QueryBuilders.boolQuery();
        //builder下有must、should以及mustNot 相当于sql中的and、or以及not

        //设置模糊搜索
        if (!StringUtils.isBlank(sku)) {
            builder.must(QueryBuilders.termQuery("sku.keyword", sku));
        }
        if (!StringUtils.isBlank(site)) {
            builder.must(QueryBuilders.termQuery("site.keyword", site));
        }

        BoolQueryBuilder purBuilder = QueryBuilders.boolQuery();
        purBuilder.should(QueryBuilders.matchPhraseQuery("order_status", "1"));
        purBuilder.should(QueryBuilders.matchPhraseQuery("order_status", "2"));
        builder.must(purBuilder);

        //构建查询
        NativeSearchQueryBuilder nativeSearchQueryBuilder = new NativeSearchQueryBuilder();
        //将搜索条件设置到构建中
        nativeSearchQueryBuilder.withQuery(builder);
        //将分页设置到构建中
        nativeSearchQueryBuilder.withPageable(pageable);

        //生产NativeSearchQuery
        NativeSearchQuery query = nativeSearchQueryBuilder.build();
        //执行,返回结果的分页
        Page<EsOrderDetail> resutlList = elasticsearchTemplate.queryForPage(query, EsOrderDetail.class, extResultMapper);
        return EsPageTool.getPageMap(resutlList, pageable);
    }

}

 

2、上面查询的映射类，在实体类上表名ES索引、类型

/**
 * 查询的es分片对应的index/type的字段
 */
@Document(indexName = "order_detail", type = "order_detail")
public class EsOrderDetail implements Serializable {
    private String order_id;
    private String order_status;
}

3、贴一下Controller接口

/**
 * ES 案例
 */
@RestController
@RequestMapping("/esDemo")
public class EsDemoController {

    private static Logger logger = LoggerFactory.getLogger(EsDemoController.class);

    @Autowired
    private EsDemoService esDemoService;

    /**
     * 查询es案例信息 （假设条件为类似sql:select order_id from sku=xx and site=xx and (order_status=1 or order_status = 2)）
     * @param jsonObject
     * @author Liujun
     * @return
     */
    @PostMapping("/selectEsDemoInfo")
    public ResponseMsg selectEsDemoInfo(@RequestBody JSONObject jsonObject) {
        try {
            Integer page = jsonObject.getInteger("page");
            Integer size = jsonObject.getInteger("size");
            String sku = jsonObject.getString("sku");
            String site = jsonObject.getString("site");

            Pageable pageable = PageRequest.of(page == null ? 0 : page-1, size == null ? 50 : size);//分页机制
            PageMap pageMap = esDemoService.selectEsDemoInfo(pageable, sku, site);
            return new ResponseMsg(Code.SUCCESS, pageMap, "根据条件查询ES案例信息成功！");
        } catch (Exception e) {
            logger.error("EsDemoController.selectEsDemoInfo 异常", e);
            return new ResponseMsg(Code.FAIL, null, "根据条件查询ES案例信息失败！"+e.getMessage());
        }
    }

    /**
     * 添加es案例信息 （数据来源为多库时 可以通到java代码实现数据插入；数据来源为同一机器时，建议通过sql+conf的形式导入数据）
     * @param pushMapList
     * @author Liujun
     * @return
     */
    @PostMapping("/saveEsDemoInfo")
    public ResponseMsg saveEsDemoInfo(@RequestBody List<Map<String, Object>> pushMapList){
        try{
            Integer saveNum = esDemoService.saveEsDemoInfo(pushMapList);
            return new ResponseMsg(Code.SUCCESS, saveNum,"添加ES案例信息成功！");
        } catch (Exception e){
            logger.error("EsDemoController.saveEsDemoInfo 异常", e);
            return new ResponseMsg(Code.FAIL,null, "添加ES案例信息失败！"+e.getMessage());
        }
    }
}

 

4、依次来看看我们的结果（这里我们通过Kibana来操作ES）

执行之前，发现ES索引是不存在的

 

调用接口，插入list集合

 

此时再查看ES索引，发现已经生成了数据

 

调用查询接口，查询到分页数据（ES查询必带分页，若不写分页代码，ES也会默认分页10条数据）