docker部署elk(elasticsearch+logstash+kibana)並監控springboot日誌

elasticsearch:

拉取鏡像：

docker pull elasticsearch:6.8.10

單機運行：

docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 elasticsearch:6.8.10

你的ip:9200，可以訪問則可以直接啓動成功，但是如果類似出現以下錯誤

[2019-10-27T14:38:59,356][INFO ][o.e.n.Node               ] [kniXCrn] starting ...
[2019-10-27T14:38:59,712][INFO ][o.e.t.TransportService   ] [kniXCrn] publish_address {172.17.0.6:9300}, bound_addresses {[::]:9300}
[2019-10-27T14:38:59,754][INFO ][o.e.b.BootstrapChecks    ] [kniXCrn] bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2019-10-27T14:38:59,827][INFO ][o.e.n.Node               ] [kniXCrn] stopping ...
[2019-10-27T14:38:59,855][INFO ][o.e.n.Node 

則修改 /etc/sysctl.conf 配置，修改如下：

打開配置文件 sysctl.conf

vim /etc/sysctl.conf 

在配置文件最下面，添加如下配置：

vm.max_map_count=655360

修改完成後，執行此命令

sysctl -p

然後，重新啓動elasticsearch，即可啓動成功；

以上是沒有指定配置文件的簡單啓動，下面介紹配置文件啓動方法：

筆者將elasticsearch的volume路徑放在/usr/local/elasticsearch下，同時創建data,conf,logs文件夾

在宿主機/usr/local/elasticsearch/conf下創建文件elasticsearch.yml

cluster.name: elk-cluster
path.data: /usr/local/elasticsearch/data
path.logs: /usr/local/elasticsearch/logs
network.host: 0.0.0.0
http.port: 9200

同時解決文件訪問權限問題：

chmod 777 /usr/local/elaticsearch/*

 

運行docker鏡像：

docker run -d --name elasticsearch \
 -v /usr/local/elasticsearch/conf/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
 -v /usr/local/elasticsearch/data:/usr/local/elasticsearch/data \
 -v /usr/local/elasticsearch/logs:/usr/local/elasticsearch/logs \
 -p 9200:9200 -p 9300:9300 elasticsearch:6.8.10

瀏覽器訪問 http://ip:9200,如果出現以下信息，則啓動成功：

 

 

logstash:

 拉取鏡像：

docker pull logstash:6.8.10

將文件掛載在宿主機/usr/local/logstash下，創建相關文件夾

在/usr/local/logstash/conf中創建文件：log4j2.properties 

logger.elasticsearchoutput.name = logstash.outputs.elasticsearch
logger.elasticsearchoutput.level = debug

然後創建一個空的配置文件：logstash.yml 

創建啓動文件：myboot.conf

input {
    tcp {
        port => 4560
        mode => "server"
        tags => ["tags"]
        codec => json_lines
    }
}
output {
    elasticsearch {
        hosts => "192.168.2.40:9200"
        index => "boot-demo-%{+YYYY.MM.dd}"
    }
}

創建配置文件：pipelines.yml 

- pipeline.id: my-logstash
  path.config: "/usr/share/logstash/config/myboot.conf"
  pipeline.workers: 3

然後就可以啓動logstash了：

docker run -d --name=logstash -p 9600:9600 -p 4560:4560 -it \
 -v /usr/local/logstash/conf/:/usr/share/logstash/config/ logstash:6.8.10

 查看日誌：

發現已經啓動成功了

 

kibana：

拉取鏡像：

docker pull kibana:6.8.10

直接啓動：這裏需要指定下elasticsearch的訪問地址

docker run -d --name kibana -e ELASTICSEARCH_URL=http://192.168.2.41:9200 -p 5601:5601 kibana:6.8.10

 訪問http://ip:5601，已經可以訪問了

 

springboot項目

由於測試項目比較簡單，測試時手動產生一些日誌，拋出一些錯誤日誌即可，這裏就直接把logback-spring.xml代碼貼出來

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>

    <springProperty scope="context" name="springAppName" source="spring.application.name"/>

    <contextName>logback</contextName>

    <!-- 彩色日誌 -->
    <!-- 彩色日誌依賴的渲染類 -->
    <conversionRule conversionWord="clr" converterClass="org.springframework.boot.logging.logback.ColorConverter" />
    <conversionRule conversionWord="wex" converterClass="org.springframework.boot.logging.logback.WhitespaceThrowableProxyConverter" />
    <conversionRule conversionWord="wEx" converterClass="org.springframework.boot.logging.logback.ExtendedWhitespaceThrowableProxyConverter" />

    <!-- 控制檯的日誌輸出樣式 -->
    <property name="CONSOLE_LOG_PATTERN"
              value="%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint}[traceId:%X{X-B3-TraceId}][spanId:%X{X-B3-SpanId}] %clr([%-5level]) %clr([pid:${PID:- }]){magenta} [%logger:%line] %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}}" />

    <property name="LOG_PATTERN"
              value="%d{yyyy-MM-dd HH:mm:ss.SSS}[traceId:%X{X-B3-TraceId}][spanId:%X{X-B3-SpanId}] [%-5level] [pid:${PID:- }]{magenta} [%logger:%line] %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}}"/>

    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder charset="utf-8">
            <pattern>${CONSOLE_LOG_PATTERN}</pattern>
            <charset>UTF-8</charset>
        </encoder>
    </appender>

    <appender name="log_stash_debug" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>192.168.2.41:4560</destination>
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>DEBUG</level>
        </filter>
        <includeCallerData>true</includeCallerData>
        <encoder charset="UTF-8" class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <timestamp>
                    <timeZone>Asia/Shanghai</timeZone>
                </timestamp>
                <!--自定義日誌輸出格式-->
                <pattern>
                    <pattern>
                        {
                        "tags": ["bootDemo-debug"],
                        "project": "myBootDemo",
                        "log_type": "log_stash_debug",
                        "level": "%level",
                        "service": "${springAppName:-}",
                        "pid": "${PID:-}",
                        "thread": "%thread",
                        "class": "%logger:%line",
                        "message": "%message",
                        "stack_trace": "%exception{20}"
                        }
                    </pattern>
                </pattern>
            </providers>
        </encoder>
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
        </encoder>
    </appender>

    <appender name="log_stash_error" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>192.168.2.41:4560</destination>
        <filter class="ch.qos.logback.classic.filter.LevelFilter">
            <level>ERROR</level>
            <onMatch>ACCEPT</onMatch>
            <onMismatch>DENY</onMismatch>
        </filter>
        <encoder charset="UTF-8" class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <timestamp>
                    <timeZone>Asia/Shanghai</timeZone>
                </timestamp>
                <pattern>
                    <pattern>
                        {
                        "tags": ["bootDemo-error"],
                        "project": "myBootDemo",
                        "log_type": "log_stash_error",
                        "level": "%level",
                        "service": "${springAppName:-}",
                        "pid": "${PID:-}",
                        "thread": "%thread",
                        "class": "%logger:%line",
                        "message": "%message",
                        "stack_trace": "%exception{20}"
                        }
                    </pattern>
                </pattern>
            </providers>
        </encoder>
        <connectionStrategy>
            <roundRobin>
                <connectionTTL>5 minutes</connectionTTL>
            </roundRobin>
        </connectionStrategy>
    </appender>

    <appender name="log_stash_business"
              class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>192.168.2.41:4560</destination>
        <encoder charset="UTF-8" class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <timestamp>
                    <timeZone>Asia/Shanghai</timeZone>
                </timestamp>
                <pattern>
                    <pattern>
                        {
                        "tags": ["bootDemo-business"],
                        "project": "myBootDemo",
                        "log_type": "log_stash_business",
                        "level": "%level",
                        "service": "${springAppName:-}",
                        "pid": "${PID:-}",
                        "thread": "%thread",
                        "class": "%logger:%line",
                        "message": "%message",
                        "stack_trace": "%exception{20}"
                        }
                    </pattern>
                </pattern>
            </providers>
        </encoder>
        <connectionStrategy>
            <roundRobin>
                <connectionTTL>5 minutes</connectionTTL>
            </roundRobin>
        </connectionStrategy>
    </appender>

    <root level="info">
        <appender-ref ref="CONSOLE" />
        <appender-ref ref="log_stash_debug"/>
        <appender-ref ref="log_stash_error"/>
    </root>

    <logger name="org.slf4j" level="info"/>
    <logger name="org.springframework" level="info"/>
    <logger name="org.apache" level="info"/>
    <logger name="pres.jeremy.testdemo" level="debug">
        <appender-ref ref="log_stash_business"/>
    </logger>
</configuration>

這裏主要收集debug，error以及其他測試的業務日誌

 

由於在logstash啓動配置文件中，輸出到es的index是按照日期區分的，在kibana控制檯中，Mangement->Kibana-> Index Patterns->reate index pattern定義elasticsearch 的index通配表達式

 

一個簡單的案例就配置完成了