Elastic：運用Elastic Stack對Kubernetes進行監控 （五）

Elastic APM是基於Elastic Stack構建的應用程序性能監視系統。 它使你可以實時監視軟件服務和應用程序-收集有關傳入請求，數據庫查詢，對緩存的調用，外部HTTP請求等的響應時間的詳細性能信息。 這樣可以輕鬆快速地找出並解決性能問題。

Elastic APM適配OpenTracing，這意味着您可以利用已經可用的大量庫來跟蹤應用程序中的組件（例如MongoDB intrumentation）。

例如，你將能夠在高度分佈式的環境（微服務體系結構）中跟蹤請求，並輕鬆快速地發現潛在的瓶頸。

Elastic APM由一個稱爲APM-Server的組件組成，用於收集跟蹤並將其發送到ElasticSearch以及與應用程序或服務一起運行的各個代理。

安裝APM服務器

我們首先需要在k8s上安裝APM-Server，以收集代理的跟蹤並將其轉發到Elasticseach。 它由用於配置設置的ConfigMap組成：

apm.configmap.yml

# apm.configmap.yml
---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: monitoring
  name: apm-server-config
  labels:
    app: apm-server
data:
  apm-server.yml: |-
    apm-server:
      host: "0.0.0.0:8200"

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}

    setup.kibana:
      host: '${KIBANA_HOST:kibana}:${KIBANA_PORT:5601}'
---

APM服務器需要公開端口8200，以允許代理轉發其跟蹤。 以下服務將該端口暴露給環境：

apm.service.yml

# apm.service.yml
---
apiVersion: v1
kind: Service
metadata:
  namespace: monitoring
  name: apm-server
  labels:
    app: apm-server
spec:
  ports:
  - port: 8200
    name: apm-server
  selector:
    app: apm-server
---

最後是 Deployment，它描述了要部署的容器：

apm.deployment.yml

# apm.deployment.yml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: monitoring
  name: apm-server
  labels:
    app: apm-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: apm-server 
  template:
    metadata:
      labels:
        app: apm-server
    spec:
      containers:
      - name: apm-server
        image: docker.elastic.co/apm/apm-server:7.6.2
        env:
        - name: ELASTICSEARCH_HOST
          value: elasticsearch-client.monitoring.svc.cluster.local
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: ELASTICSEARCH_USERNAME
          value: elastic
        - name: ELASTICSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              name: elasticsearch-pw-elastic
              key: password
        - name: KIBANA_HOST
          value: kibana.monitoring.svc.cluster.local
        - name: KIBANA_PORT
          value: "5601"
        ports:
        - containerPort: 8200
          name: apm-server
        volumeMounts:
        - name: config
          mountPath: /usr/share/apm-server/apm-server.yml
          readOnly: true
          subPath: apm-server.yml
      volumes:
      - name: config
        configMap:
          name: apm-server-config
---

現在，我們可以部署堆棧的這個新組件：

kubectl apply  -f apm.configmap.yml \
                 -f apm.service.yml \
                 -f apm.deployment.yml

$ kubectl apply  -f apm.configmap.yml \
>                  -f apm.service.yml \
>                  -f apm.deployment.yml
configmap/apm-server-config created
service/apm-server created
deployment.apps/apm-server created

檢查一切是否正常運行：

 kubectl get all -n monitoring -l app=apm-server

$  kubectl get all -n monitoring -l app=apm-server
NAME                              READY   STATUS    RESTARTS   AGE
pod/apm-server-58b6b7d8d9-jk8r9   1/1     Running   0          43s


NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/apm-server   ClusterIP   10.108.187.142   <none>        8200/TCP   43s


NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/apm-server   1/1     1            1           43s

NAME                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/apm-server-58b6b7d8d9   1         1         1       43s

現在，我們可以在Spring-Boot應用程序上安裝代理。

在Spring-Boot應用上安裝Java agent

在本文的最後一部分中，我們將在示例應用程序spring-boot-simple上配置Elastic APM Java agent。

首先，我們需要將jar elastic-apm-agent-1.8jar放入容器中。 添加以下行以在Docker構建映像時下載代理JAR。

RUN wget -O /apm-agent.jar https://search.maven.org/remotecontent?filepath=co/elastic/apm/elastic-apm-agent/1.8.0/elastic-apm-agent-1.8.0.jar

Dockfile:

FROM openjdk:8-jdk-alpine

COPY target/spring-boot-simple.jar /app.jar

RUN wget -O /apm-agent.jar https://search.maven.org/remotecontent?filepath=co/elastic/apm/elastic-apm-agent/1.8.0/elastic-apm-agent-1.8.0.jar

CMD java -jar /app.jar

其次，將以下依賴項添加到您的應用程序中，以便您能夠集成開放式跟蹤庫（瞭解更多）和/或使用Elastic APM API手動檢測某些組件（瞭解更多）。

        <dependency>
            <groupId>co.elastic.apm</groupId>
            <artifactId>apm-agent-api</artifactId>
            <version>${elastic-apm.version}</version>
        </dependency>
        <dependency>
            <groupId>co.elastic.apm</groupId>
            <artifactId>apm-opentracing</artifactId>
            <version>${elastic-apm.version}</version>
        </dependency>
        <dependency>
            <groupId>io.opentracing.contrib</groupId>
            <artifactId>opentracing-spring-cloud-mongo-starter</artifactId>
            <version>${opentracing-spring-cloud.version}</version>
        </dependency>

然後，我們將更改Deployment，以在啓用Java代理並將其連接到APM服務器的情況下啓動Spring-Boot應用程序。

# spring-boot-simple.deployment.yml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: default
  name: spring-boot-simple
  labels:
    app: spring-boot-simple
spec:
  replicas: 1
  selector:
    matchLabels:
      app: spring-boot-simple
  template:
    metadata:
      labels:
        app: spring-boot-simple
    spec:
      containers:
      - image: gjeanmart/spring-boot-simple:0.0.1-SNAPSHOT
        imagePullPolicy: Always
        name: spring-boot-simple
        command:
          - "java"
          - "-javaagent:/apm-agent.jar"
          - "-Delastic.apm.active=$(ELASTIC_APM_ACTIVE)"
          - "-Delastic.apm.server_urls=$(ELASTIC_APM_SERVER)"
          - "-Delastic.apm.service_name=spring-boot-simple"
          - "-jar"
          - "app.jar"
        env:
          - name: SPRING_DATA_MONGODB_HOST
            value: mongo
          - name: ELASTIC_APM_ACTIVE
            value: "true"
          - name: ELASTIC_APM_SERVER
            value: http://apm-server.monitoring.svc.cluster.local:8200
        ports:
        - containerPort: 8080
---

現在重新應用部署，並且spring-boot-simple應該重新啓動：

 kubectl apply -f spring-boot-simple.yml

然後，我們執行像在第一篇文章中介紹的那樣：

kubectl get svc

上面顯示所有的service:

$ kubectl get svc
NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes           ClusterIP   10.96.0.1       <none>        443/TCP          10h
mongo                ClusterIP   10.98.1.3       <none>        27017/TCP        9h
spring-boot-simple   NodePort    10.101.241.38   <none>        8080:30202/TCP   9h

我們通過如下的方式來找到spring-boot-simple的服務地址：

 minikube service spring-boot-simple --url

$ minikube service spring-boot-simple --url
🏃  Starting tunnel for service spring-boot-simple.
|-----------|--------------------|-------------|------------------------|
| NAMESPACE |        NAME        | TARGET PORT |          URL           |
|-----------|--------------------|-------------|------------------------|
| default   | spring-boot-simple |             | http://127.0.0.1:62891 |
|-----------|--------------------|-------------|------------------------|
http://127.0.0.1:62891
❗  Because you are using docker driver on Mac, the terminal needs to be open to run it.

get messages

curl -X GET http://YourIP:YourPort/message

get messages (slow request)

使用屬性sleep = <ms>可以減慢請求的速度。

curl -X GET http://YourIP:YourPort/message?sleep=3000

get messages (error)

curl -X GET http://YourIP:YourPort/message?error=true

針對我的情況就是：

$ curl http://127.0.0.1:62891
Greetings from Spring Boot!liuxg:minikube liuxg$ 

liuxg:minikube liuxg$ curl http://127.0.0.1:62891
Greetings from Spring Boot!

liuxg:minikube liuxg$ curl http://127.0.0.1:62891
Greetings from Spring Boot!

liuxg:minikube liuxg$ curl http://127.0.0.1:62891/message?sleep=300
[{"id":"5eace24730c49e000198b8d6","message":"hello+world=","postedAt":"2020-05-02T03:00:23.860+0000"}]

liuxg:minikube liuxg$ curl http://127.0.0.1:62891/message?sleep=5000
[{"id":"5eace24730c49e000198b8d6","message":"hello+world=","postedAt":"2020-05-02T03:00:23.860+0000"}]

liuxg:minikube liuxg$ curl http://127.0.0.1:62891/message?error=true
{"timestamp":"2020-05-02T12:39:07.913+0000","status":500,"error":"Internal Server Error","message":"java.lang.Exception: Random error","path":"/message"}

liuxg:minikube liuxg$ curl http://127.0.0.1:62891/message?error=true
{"timestamp":"2020-05-02T12:39:12.307+0000","status":500,"error":"Internal Server Error","message":"java.lang.Exception: Random error","path":"/message"}

我們打開Kibana APM應用：

我們可以看到在Kibana中有一個叫做spring-boot-simple的Service。我們點擊上面的超鏈接：

我們可以看到裏面的transactions。點擊其中的getMessages:

 

它詳細地顯示了我們快的和慢的transaction所花的時間在哪裏。我們也可以點擊Error來查看erros：

我們可以看到Error的詳細的信息。