elasticsearch集羣bulk操作測試

服務器信息：

(虛擬機) * 3

cpu:

-------------------------------------------------------

總核數 = 物理CPU個數 X 每顆物理CPU的核數

總邏輯CPU數 = 物理CPU個數 X 每顆物理CPU的核數 X 超線程數

-------------------------------------------------------

物理CPU個數

8

-------------------------------------------------------

查看每個物理CPU中core的個數(即核數)

cpu cores : 1

-------------------------------------------------------

查看邏輯CPU的個數

8

-------------------------------------------------------

查看CPU信息（型號）

8 Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz

------------------------------------------------------

free:

total used free shared buff/cache available

Mem: 15G 2.3G 10G 9.0M 3.2G 12G

Swap: 0B 0B 0B

disk:

機械磁盤 西部數據紫盤5400

數據來源Kafka 

TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID

first_floor 3 172554612 268659420 96104808 - - -

first_floor 4 172468166 268664997 96196831 - - -

first_floor 5 170698697 266998946 96300249 - - -

first_floor 6 169190488 265599032 96408544 - - -

first_floor 0 182175112 278532188 96357076 - - -

first_floor 1 43120790 270504406 227383616 - - -

first_floor 2 174732636 270500824 95768188 - - -

測試一：

elasticsearch 存儲 2w/s

開始時間	結束時間	歷時(s)	總數據	平均值(/s)
		112	2471491	22000

index：

分片數5 無備份 refreshInterval = "10s"

JAVA程序Consumer配置：

@Bean("kafkaListenerEsContainerFactory")
public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String,String>> kafkaListenerEsContainerFactory(){
    ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();

    factory.setConsumerFactory(esConsumerFactory());
    // 併發數量 等於消費者數量
    factory.setConcurrency(7);
    // 批量獲取
    factory.setBatchListener(true);
    factory.getContainerProperties().setPollTimeout(3000);
    factory.setAutoStartup(true);
    return factory;

}

public ConsumerFactory<String, String> esConsumerFactory() {
    return new DefaultKafkaConsumerFactory<>(esConsumerConfigs());
}

public Map<String, Object> esConsumerConfigs() {
    Map<String, Object> propsMap = new HashMap<>();
    propsMap.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, servers);
    propsMap.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, enableAutoCommit);
    propsMap.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, autoCommitInterval);
    propsMap.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    propsMap.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class);
    propsMap.put(ConsumerConfig.GROUP_ID_CONFIG, taskGroupId);
    propsMap.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, autoOffsetReset);
    //最多批量獲取15000個
    propsMap.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG,15000);
    return propsMap;
}

@Bean
public KafkaProperties.Listener listener() {
    return new KafkaProperties.Listener();
}

JAVA程序BulkProcessor配置：

@Bean
    public BulkProcessor bulkProcessor() throws UnknownHostException {
        final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(AuthScope.ANY,new UsernamePasswordCredentials("elastic","Waner123456"));
        Settings settings = Settings.builder()
                .put("cluster.name", "waner-es-cluster")
                .put("client.transport.sniff",true)
                .build();
        TransportClient transportClient = new PreBuiltTransportClient(settings);
//        TransportAddress transportAddress0 = new TransportAddress(Inet4Address.getByName("172.168.1.210"), 9300);
//        TransportAddress transportAddress1 = new TransportAddress(Inet4Address.getByName("172.168.1.211"), 9300);
        TransportAddress transportAddress2 = new TransportAddress(Inet4Address.getByName("172.168.1.212"), 9300);
        TransportAddress transportAddress3 = new TransportAddress(Inet4Address.getByName("172.168.1.213"), 9300);
        TransportAddress transportAddress4 = new TransportAddress(Inet4Address.getByName("172.168.1.214"), 9300);
        transportClient.addTransportAddresses(transportAddress2,transportAddress3,transportAddress4);
        final Long[] start = {0L};
        return BulkProcessor.builder(transportClient, new BulkProcessor.Listener() {
            @Override
            public void beforeBulk(long executionId, BulkRequest request) {
                start[0] = System.currentTimeMillis();
                LOGGER.info("批量執行 [{}] with {} 請求 {}", executionId, executionId,start);
            }
            @Override
            public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {
                if(response.hasFailures()){
                    LOGGER.error("Bulk [{}] 執行失敗,響應 = {}", executionId, response.buildFailureMessage());
                } else {
                    long end = System.currentTimeMillis();
                    LOGGER.info("Bulk [{}] 請求數據 {} 完成於 {} milliseconds 用時 {}", executionId,request.numberOfActions(), response.getTook().getMillis(),(end-start[0]));
                }
                BulkItemResponse[] responses = response.getItems();

            }
            @Override
            public void afterBulk(long executionId, BulkRequest request, Throwable failure) {
                LOGGER.error("{} 批量數據失敗, 原因 :{}", request.numberOfActions(), failure);
            }
        }).setBulkActions(20000)  //每2w條執行一次bulk插入
                .setBulkSize(new ByteSizeValue(17, ByteSizeUnit.MB)) //  數據量達到17M後執行bulk插入
                .setFlushInterval(TimeValue.timeValueSeconds(20)) //無論數據量多少，間隔20s執行一次bulk
                .setConcurrentRequests(10) // 允許併發的bulk請求數
                .setBackoffPolicy(BackoffPolicy.exponentialBackoff(TimeValue.timeValueMillis(100), 3))
                .build();
    }

性能監控：

heap偶爾偏高 cpu load一切正常

 

 

測試二：

elasticsearch 存儲 1.9w/s

開始時間	結束時間	歷時(s)	總數據	平均值(/s)
1575341482800	1575342496978	172	3437795	19987.18

index：

分片數5 無備份 refreshInterval = "10s"

JAVA程序Consumer配置：

@Bean("kafkaListenerEsContainerFactory")
public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String,String>> kafkaListenerEsContainerFactory(){
    ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();

    factory.setConsumerFactory(esConsumerFactory());
    // 併發數量 等於消費者數量
    factory.setConcurrency(7);
    // 批量獲取
    factory.setBatchListener(true);
    factory.getContainerProperties().setPollTimeout(3000);
    factory.setAutoStartup(true);
    return factory;

}

public ConsumerFactory<String, String> esConsumerFactory() {
    return new DefaultKafkaConsumerFactory<>(esConsumerConfigs());
}

public Map<String, Object> esConsumerConfigs() {
    Map<String, Object> propsMap = new HashMap<>();
    propsMap.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, servers);
    propsMap.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, enableAutoCommit);
    propsMap.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, autoCommitInterval);
    propsMap.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    propsMap.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class);
    propsMap.put(ConsumerConfig.GROUP_ID_CONFIG, taskGroupId);
    propsMap.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, autoOffsetReset);
    //最多批量獲取20000個
    propsMap.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG,20000);
    return propsMap;
}

@Bean
public KafkaProperties.Listener listener() {
    return new KafkaProperties.Listener();
}

JAVA程序BulkProcessor配置：

@Bean
    public BulkProcessor bulkProcessor() throws UnknownHostException {
        final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(AuthScope.ANY,new UsernamePasswordCredentials("elastic","Waner123456"));
        Settings settings = Settings.builder()
                .put("cluster.name", "waner-es-cluster")
                .put("client.transport.sniff",true)
                .build();
        TransportClient transportClient = new PreBuiltTransportClient(settings);
//        TransportAddress transportAddress0 = new TransportAddress(Inet4Address.getByName("172.168.1.210"), 9300);
//        TransportAddress transportAddress1 = new TransportAddress(Inet4Address.getByName("172.168.1.211"), 9300);
        TransportAddress transportAddress2 = new TransportAddress(Inet4Address.getByName("172.168.1.212"), 9300);
        TransportAddress transportAddress3 = new TransportAddress(Inet4Address.getByName("172.168.1.213"), 9300);
        TransportAddress transportAddress4 = new TransportAddress(Inet4Address.getByName("172.168.1.214"), 9300);
        transportClient.addTransportAddresses(transportAddress2,transportAddress3,transportAddress4);
        final Long[] start = {0L};
        return BulkProcessor.builder(transportClient, new BulkProcessor.Listener() {
            @Override
            public void beforeBulk(long executionId, BulkRequest request) {
                start[0] = System.currentTimeMillis();
                LOGGER.info("批量執行 [{}] with {} 請求 {}", executionId, executionId,start);
            }
            @Override
            public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {
                if(response.hasFailures()){
                    LOGGER.error("Bulk [{}] 執行失敗,響應 = {}", executionId, response.buildFailureMessage());
                } else {
                    long end = System.currentTimeMillis();
                    LOGGER.info("Bulk [{}] 請求數據 {} 完成於 {} milliseconds 用時 {}", executionId,request.numberOfActions(), response.getTook().getMillis(),(end-start[0]));
                }
                BulkItemResponse[] responses = response.getItems();

            }
            @Override
            public void afterBulk(long executionId, BulkRequest request, Throwable failure) {
                LOGGER.error("{} 批量數據失敗, 原因 :{}", request.numberOfActions(), failure);
            }
        }).setBulkActions(30000)  //每2w條執行一次bulk插入
                .setBulkSize(new ByteSizeValue(20, ByteSizeUnit.MB)) //  數據量達到20M後執行bulk插入
                .setFlushInterval(TimeValue.timeValueSeconds(20)) //無論數據量多少，間隔20s執行一次bulk
                .setConcurrentRequests(10) // 允許併發的bulk請求數
                .setBackoffPolicy(BackoffPolicy.exponentialBackoff(TimeValue.timeValueMillis(100), 3))
                .build();

以上實際每次插入數據峯值爲17300/bulk，可以適當上調bulk_size大小

性能監控：

heap偶爾偏高 cpu load一切正常