微服務鏈路追蹤組件Skywalking實戰

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"1. skywalking是什麼","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於一個大型的幾十個、幾百個微服務構成的微服務架構系統,通常會遇到下面一些問題,比如:","attrs":{}}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"如何串聯整個調用鏈路,快速定位問題?","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"如何理清各個微服務之間的依賴關係?","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"如何進行各個微服務接口的性能分折?","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"如何跟蹤整個業務流程的調用處理順序?","attrs":{}}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"skywalking是一個國產開源框架,2015年由吳晟開源 , 2017年加入Apache孵化器。skywalking是分佈式系統的應用程序性能監視工具,專爲微服務、雲原生架構和基於容器(Docker、K8s、Mesos)架構而設計。SkyWalking 是觀察性分析平臺和應用性能管理系統,提供分佈式追蹤、服務網格遙測分析、度量聚合和可視化一體化解決方案。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"官網:","attrs":{}},{"type":"link","attrs":{"href":"http://skywalking.apache.org/","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"http://skywalking.apache.org/","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下載:","attrs":{}},{"type":"link","attrs":{"href":"http://skywalking.apache.org/downloads/","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"http://skywalking.apache.org/downloads/","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"調用鏈選型","attrs":{}}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"Zipkin是Twitter開源的調用鏈分析工具,目前基於springcloud sleuth得到了廣泛的使用,特點是輕量,使用部署簡單。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"Pinpoint是韓國人開源的基於字節碼注入的調用鏈分析,以及應用監控分析工具。特點是支持多種插件,UI功能強大,接入端無代碼侵入。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"SkyWalking是本土開源的基於字節碼注入的調用鏈分析,以及應用監控分析工具。特點是支持多種插件,UI功能較強,接入端無代碼侵入。目前已加入Apache孵化器。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"CAT是大衆點評開源的基於編碼和配置的調用鏈分析,應用監控分析,日誌採集,監控報警等一系列的監控平臺工具。","attrs":{}}]}]}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/af/af6f457421205df66551f9818bd1aee9.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"探針性能對比","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"模擬了三種併發用戶:500,750,1000。使用jmeter測試,每個線程發送30個請求,設置思考時間爲10ms。使用的採樣率爲1,即100%,這邊與生產可能有差別。pinpoint默認的採樣率爲20,即50%,通過設置agent的配置文件改爲100%。zipkin默認也是1。組合起來,一共有12種。下面看下彙總表:","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5f/5f35112cec220b92a4469d1b20ec3cef.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從上表可以看出,在三種鏈路監控組件中,","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"skywalking的探針對吞吐量的影響最小,zipkin的吞吐量居中。pinpoint的探針對吞吐量的影響較爲明顯","attrs":{}},{"type":"text","text":",在500併發用戶時,測試服務的吞吐量從1385降低到774,影響很大。然後再看下CPU和memory的影響,在內部服務器進行的壓測,對CPU和memory的影響都差不多在10%之內。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"1.1 Skywalking主要功能特性","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、多種監控手段,可以通過語言探針和service mesh獲得監控的數據;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、支持多種語言自動探針,包括 Java,.NET Core 和 Node.JS;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3、輕量高效,無需大數據平臺和大量的服務器資源;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4、模塊化,UI、存儲、集羣管理都有多種機制可選;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"5、支持告警;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"6、優秀的可視化解決方案;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"1.2 Skywalking整體架構","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/dd/dd27023692ccf78f93ffe8f63c4a61b2.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"整個架構分成四部分:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、上部分Agent :負責從應用中,收集鏈路信息,發送給 SkyWalking OAP 服務器;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、下部分 SkyWalking OAP :負責接收Agent發送的Tracing數據信息,然後進行分析(Analysis Core),存儲到外部存儲器(Storage),最終提供查詢(Query)功能;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3、右部分Storage:Tracing數據存儲,目前支持ES、MySQL、Sharding Sphere、TiDB、H2多種存儲器,目前採用較多的是ES,主要考慮是SkyWalking開發團隊自己的生產環境採用ES爲主;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4、左部分SkyWalking UI:負責提供控制檯,查看鏈路等等;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"SkyWalking支持三種探針:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"● Agent – 基於ByteBuddy字節碼增強技術實現,通過jvm的agent參數加載,並在程序啓動時攔截指定的方法來收集數據。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"● SDK – 程序中顯式調用SkyWalking提供的SDK來收集數據,對應用有侵入。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"● Service Mesh – 通過Service mesh的網絡代理來收集數據。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"後端(Backend)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接受探針發送過來的數據,進行度量分析,調用鏈分析和存儲。後端主要分爲兩部分:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"● OAP(Observability Analysis Platform)- 進行度量分析和調用鏈分析的後端平臺,並支持將數據存儲到各種數據庫中,如:ElasticSearch,MySQL,InfluxDB等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"● OAL(Observability Analysis Language)- 用來進行度量分析的DSL,類似於SQL,用於查詢度量分析結果和警報。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"界面(UI)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"● RocketBot UI – SkyWalking 7.0.0 的默認web UI","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"● CLI – 命令行界面","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這三個模塊的交互流程:","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/6c/6ce00717759acbddc5d937be6d112d09.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"1.3 SkyWalking 環境搭建部署","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e6/e6beab8dd3d0e37ac53531b10d8f0248.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"skywalking agent和業務系統綁定在一起,負責收集各種監控數據","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Skywalking oapservice是負責處理監控數據的,比如接受skywalking agent的監控數據,並存儲在數據庫中;接受skywalking webapp的前端請求,從數據庫查詢數據,並返回數據給前端。Skywalking oapservice通常以集羣的形式存在。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"skywalking webapp,前端界面,用於展示數據。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用於存儲監控數據的數據庫,比如mysql、elasticsearch等。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.3.1 下載 SkyWalking","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下載:","attrs":{}},{"type":"link","attrs":{"href":"http://skywalking.apache.org/downloads/","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"http://skywalking.apache.org/downloads/","attrs":{}}]}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4d/4ddb4f2d3341a3f674f99ca3eeb022cb.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"目錄結構","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/87/8775b7e4107f2b99f72ae031e5bd6107.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.3.2 搭建SkyWalking OAP 服務","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"先使用默認的H2數據庫存儲,不用修改配置","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"config/application.yml","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f6/f6bded59bc7a73d63a1e0f6199670fee.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"啓動腳本bin/startup.sh","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/0c/0c2fefc28514340e57dbe7b80e0694fb.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"日誌信息存儲在logs目錄","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/0a/0a8e7cc84df31484384a2146af088e6c.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"啓動成功後會啓動兩個服務,一個是skywalking-oap-server,一個是skywalking-web-ui","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"skywalking-oap-server服務啓動後會暴露11800 和 12800 兩個端口,分別爲收集監控數據的端口11800和接受前端請求的端口12800,修改端口可以修改config/applicaiton.yml","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2c/2cb177091cf5028a082bc6d1b4bc0d0a.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"skywalking-web-ui服務會佔用 8080 端口, 修改端口可以修改webapp/webapp.yml","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c8/c87bd436526613f8711f0e2f4a5d9c7c.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"server.port:SkyWalking UI服務端口,默認是8080;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"collector.ribbon.listOfServers:SkyWalking OAP服務地址數組,SkyWalking UI界面的數據是通過請求SkyWalking OAP服務來獲得;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"訪問:","attrs":{}},{"type":"link","attrs":{"href":"http://192.168.3.14:8080/","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"http://192.168.3.100:8080/","attrs":{}}]}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9f/9f309d74fbbfe0437099a370026373bd.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"頁面的右下角可以中英文切換,可以切換選擇要展示的時間區間的跟蹤數據。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"1.4 SkyWalking中三個概念","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"服務(Service) :","attrs":{}},{"type":"text","text":"表示對請求提供相同行爲的一系列或一組工作負載,在使用Agent時,可以定義服務的名字;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"服務實例(Service Instance) :","attrs":{}},{"type":"text","text":"上述的一組工作負載中的每一個工作負載稱爲一個實例, 一個服務實例實際就是操作系統上的一個真實進程;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"端點(Endpoint) :","attrs":{}},{"type":"text","text":"對於特定服務所接收的請求路徑, 如HTTP的URI路徑和gRPC服務的類名 + 方法簽名;","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b5/b521877b82266fe8bf854e7d1e8f99fd.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"2. SkyWalking快速開始","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2.1 SkyWalking Agent跟蹤微服務","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.1.1 通過jar包方式接入","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"準備一個springboot程序,打成可執行jar包,寫一個shell腳本,在啓動項目的Shell腳本上,通過 -javaagent 參數進行配置SkyWalking Agent來跟蹤微服務;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"startup.sh腳本:","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"#!/bin/sh\n# SkyWalking Agent配置\nexport SW_AGENT_NAME=springboot-skywalking-demo #Agent名字,一般使用`spring.application.name`\nexport SW_AGENT_COLLECTOR_BACKEND_SERVICES=127.0.0.1:11800 #配置 Collector 地址。\nexport SW_AGENT_SPAN_LIMIT=2000 #配置鏈路的最大Span數量,默認爲 300。\nexport JAVA_AGENT=-javaagent:/usr/local/soft/apache-skywalking-apm-bin-es7/agent/skywalking-agent.jar\njava $JAVA_AGENT -jar springboot-skywalking-demo-0.0.1-SNAPSHOT.jar #jar啓動","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"啓動日誌","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/45/45e1daa16756888f2a400af4f9facf76.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"等同於","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"java -javaagent:/usr/local/soft/apache-skywalking-apm-bin-es7/agent/skywalking-agent.jar \n-DSW_AGENT_COLLECTOR_BACKEND_SERVICES=127.0.0.1:11800 \n-DSW_AGENT_NAME=springboot-skywalking-demo -jar springboot-skywalking-demo-0.0.1-SNAPSHOT.jar","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"參數名對應agent/config/agent.config配置文件中的屬性。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"屬性對應的源碼:org.apache.skywalking.apm.agent.core.conf.Config.java","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"# The service name in UI\nagent.service_name=${SW_AGENT_NAME:Your_ApplicationName}\n# Backend service addresses.\ncollector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:127.0.0.1:11800}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們也可以使用skywalking.+配置文件中的配置名作爲系統配置項來進行覆蓋。 javaagent參數配置方式優先級更高","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"-javaagent:D:\\apache\\apache-skywalking-apm-es7-8.4.0\\apache-skywalking-apm-bin-es7\\agent\\skywalking-agent.jar\n-Dskywalking.agent.service_name=springboot-skywalking-demo\n-Dskywalking.collector.backend_service=192.168.3.100:11800","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/da/da403b40e460c23606e3cf7bd945151a.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"測試: ","attrs":{}},{"type":"link","attrs":{"href":"http://192.168.3.100:8000/user/list","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"http://192.168.3.100:8000/user/list","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在啓動程序前加一個-javaagent 參數即可完成對程序的跟蹤","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/de/de407ca7bf705a2f19ae074fa78c260e.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.1.2 在IDEA中使用Skywalking","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在運行的程序配置jvm參數,如下圖所示:","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/14/14ad1b7f33e37459bde6e353aeee0f9a.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"# skywalking-agent.jar的本地磁盤的路徑\n-javaagent:D:\\apache\\apache-skywalking-apm-es7-8.4.0\\apache-skywalking-apm-bin-es7\\agent\\skywalking-agent.jar\n# 在skywalking上顯示的服務名\n-DSW_AGENT_NAME=springboot-skywalking-demo\n# skywalking的collector服務的IP及端口\n-DSW_AGENT_COLLECTOR_BACKEND_SERVICES=192.168.3.100:11800 ","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.1.3 Skywalking跨多個微服務跟蹤","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Skywalking跨多個微服務跟蹤,只需要每個微服務啓動時添加javaagent參數即可。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"測試:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"啓動微服務mall-gateway,mall-order,mall-user ,配置skywalking的jvm參數","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"http://localhost:8888/user/findOrderByUserId/1","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"http://localhost:8888/user/findOrderByUserId/1","attrs":{}}]}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8c/8cb63b4a2346bb006a6603d6f1af5bf8.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e9/e97b5ffe95ca638f02a1f2dcb801c247.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"注意:此處存在bug,跟蹤鏈路不顯示gateway","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"拷貝agent/optional-plugins目錄下的gateway插件到agent/plugins目錄","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b4/b49fc2bfba10267afd2994c00f43245e.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c4/c4edc17104615bf08695ad3d5d7503c2.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/64/643e21438c8690d0bcd5b0ce27916867.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e9/e90e62ddbdb75fce19d5cdb26e1ae660.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2.2 Skywalking告警通知","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"skywalking告警的核心由一組規則驅動,這些規則定義在config/alarm-settings.yml文件中,告警規則的定義分爲三部分:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、告警規則:它們定義了應該如何觸發度量警報,應該考慮什麼條件;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、網絡鉤子(Webhook}:當警告觸發時,哪些服務終端需要被通知;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3、gRPC鉤子:遠程gRPC方法的主機和端口,告警觸發後調用;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了方便,skywalking發行版中提供了默認的alarm-setting.yml文件,包括一些規則,每個規則有英文註釋,可以根據註釋得知每個規則的作用:","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在最近10分鐘的3分鐘內服務平均響應時間超過1000ms","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最近10分鐘內,服務成功率在2分鐘內低於80%","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務實例的響應時間在過去10分鐘的2分鐘內超過1000ms","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據庫訪問{name}的響應時間在過去10分鐘的2分鐘內超過1000ms","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"只要我們的服務請求符合alarm-setting.yml文件中的某一條規則就會觸發告警。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"比如service_resp_time_rule規則:","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b2/b20f3371a9fb7aeee51116ae791eb589.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該規則表示服務{name}的響應時間在最近10分鐘的3分鐘內超過1000ms;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"測試:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"編寫接口,模擬慢查詢","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"@RequestMapping(\"/info/{id}\")\npublic User info(@PathVariable(\"id\") Integer id){\n\n try {\n Thread.sleep(2000);\n } catch (InterruptedException e) {\n e.printStackTrace();\n }\n\n return userService.getById(id);\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"回調接口","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"@RequestMapping(\"/notify\")\npublic String notify(@RequestBody Object obj){\n //TODO 告警信息,給技術負責人發短信,釘釘消息,郵件,微信通知等\n System.err.println(obj.toString());\n return \"notify successfully\";\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在config/alarm-settings.yml中配置回調接口,並重啓skywalking服務","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/17/17880a5fd14b9927ed628ebad68d989e.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"測試訪問:","attrs":{}},{"type":"link","attrs":{"href":"http://localhost:8000/user/info/1","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"http://localhost:8000/user/info/1","attrs":{}}]},{"type":"text","text":",滿足告警規則後,控制檯輸出告警信息","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/7e/7ef1ea6f81e2a754ef1ec94e00aa6d6d.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SkyWalking UI顯示告警信息","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b8/b8849da3b6e978ff8356d210b60ba24d.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"參考: ","attrs":{}},{"type":"link","attrs":{"href":"https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-alarm.md","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-alarm.md","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對接釘釘:","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5b/5b503e1cd3394412f61f2145e3560bb7.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Webhook回調通知","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SkyWalking告警Webhook回調要求接收方是一個Web容器(比如tomcat服務),告警的消息會通過HTTP請求進行發送, 請求方法爲POST, Content-Type爲application/json, JSON格式基於List的集合對象數據, 集合中的每個AlarmMessage包含以下信息:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、scopeId. 所有可用的Scope,參考:org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、name. 目標 Scope 的實體名稱;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3、id0. Scope 實體的 ID;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4、id1. 未使用;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"5、ruleName. 在 alarm-settings.yml 中配置的規則名;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"6、alarmMessage. 報警消息內容;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"7、startTime. 告警時間, 位於當前時間與 UTC 1970/1/1 之間;","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"[{\n\tscopeId = 2,\n\tscope = SERVICE_INSTANCE,\n\tname = 98e1839 a6fdf48b0aedb0ecabb8ea5f7 @192 .168 .233 .1 of springboot - skywalking - demo,\n\tid0 = c3ByaW5nYm9vdC1za3l3YWxraW5nLWRlbW8 = .1 _OThlMTgzOWE2ZmRmNDhiMGFlZGIwZWNhYmI4ZWE1ZjdAMTkyLjE2OC4yMzMuMQ == ,\n\tid1 = ,\n\truleName = service_instance_resp_time_rule,\n\talarmMessage = Response time of service instance 98e1839 a6fdf48b0aedb0ecabb8ea5f7 @192 .168 .233 .1 of springboot - skywalking - demo is more than 1000 ms in 2 minutes of last 10 minutes,\n\tstartTime = 1613913565462\n}, {\n\tscopeId = 6,\n\tscope = ENDPOINT_RELATION,\n\tname = User in User to / user / info / {\n\t\tid\n\t} in springboot - skywalking - demo,\n\tid0 = VXNlcg == .0 _VXNlcg == ,\n\tid1 = c3ByaW5nYm9vdC1za3l3YWxraW5nLWRlbW8 = .1 _L3VzZXIvaW5mby97aWR9,\n\truleName = endpoint_relation_resp_time_rule,\n\talarmMessage = Response time of endpoint relation User in User to / user / info / {\n\t\tid\n\t} in springboot - skywalking - demo is more than 1000 ms in 2 minutes of last 10 minutes,\n\tstartTime = 1613913565462\n}]","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2.3 Skywalking持久化跟蹤數據","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.3.1 基於mysql持久化","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1. 修改config目錄下的application.yml,使用mysql作爲持久化存儲的倉庫","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d9/d9488ca6c0cd8321e6e13a3a99e08586.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2. 修改mysql連接配置","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/0d/0dc9a26e0298058e04c4acbed5dbf721.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"storage:\n #選擇使用mysql 默認使用h2,不會持久化,重啓skyWalking之前的數據會丟失\n selector: ${SW_STORAGE:mysql}\n #使用mysql作爲持久化存儲的倉庫\n mysql:\n properties:\n #數據庫連接地址\n jdbcUrl: ${SW_JDBC_URL:\"jdbc:mysql://1ocalhost:3306/swtest\"}\n #用戶名\n dataSource.user: ${SW_DATA_SOURCE_USER:root}\n #密碼\n dataSource.password: ${SW_DATA_SOURCE_PASSWORD:root}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"注意:需要添加mysql數據驅動包,因爲在lib目錄下是沒有mysql數據驅動包的,所以修改完配置啓動是會報錯,啓動失敗的。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/05/05fb9fe26df1cc51d84812c0fa5bb84a.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3. 添加mysql數據驅動包到oap-libs目錄下","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/0d/0dbf88668786f22a2cf73981e6fc002d.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4. 啓動Skywalking","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/50/505f1fed61ebc70086c003f608ee1a15.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"查看swtest數據庫,可以看到生成了很多表。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9a/9a919849fc0e483991c905312893bd6f.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"說明啓動成功了,打開配置對應的地址","attrs":{}},{"type":"link","attrs":{"href":"http://192.168.3.14:8080/","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"http://192.168.3.100:8080/","attrs":{}}]},{"type":"text","text":",可以看到skywalking的web界面。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2c/2c57867ad210ba61224f0640fdf611a6.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"測試:重啓skywalking,驗證跟蹤數據會不會丟失","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.3.2 基於elasticsearch持久化","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.準備好elasticsearch環境","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"啓動elasticsearch服務","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"su - es\ncd /usr/local/soft/elasticsearch-7.6.1/\nbin/elasticsearch -d","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/63/63555f7b9a60a7179993f6a9208dacd8.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"啓動elasticsearch-head服務,訪問","attrs":{}},{"type":"link","attrs":{"href":"http://192.168.3.100:9100","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"http://192.168.3.100:9100","attrs":{}}]}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"cd /usr/local/soft/elasticsearch-head/node_modules/grunt\nnohup bin/grunt server &","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/cd/cd330df656b1b0635b62c7676506799f.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2.修改config/application.yml配置文件","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/39/3973ed991f44363f8bc0f01885f71820.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"修改elasticsearch7的連接配置","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/98/98d1689d4b6384288906e07c206fa4f4.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3. 啓動Skywalking服務","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e6/e6a9c8653e337f227a05c071ab868c05.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"啓動時會向elasticsearch中創建大量的index索引用於持久化數據,每天會產生一個新的索引文件。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"測試:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"啓動應用程序,查看跟蹤數據是否已經持久化到elasticsearch的索引中,然後重啓skywalking,驗證跟蹤數據會不會丟失","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/7d/7d80785a2444a7833bd04b7af6401aae.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2.4 自定義SkyWalking鏈路追蹤","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果我們希望對項目中的業務方法,實現鏈路追蹤,方便我們排查問題,可以使用如下的代碼","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"引入依賴","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\n\n org.apache.skywalking\n apm-toolkit-trace\n 8.4.0\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在業務方法中可以TraceContext獲取到traceId","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"@RequestMapping(\"/list\")\npublic List list(){\n\n //TraceContext可以綁定key-value\n TraceContext.putCorrelation(\"name\", \"fox\");\n Optional op = TraceContext.getCorrelation(\"name\");\n log.info(\"name = {} \", op.get());\n //獲取跟蹤的traceId\n String traceId = TraceContext.traceId();\n log.info(\"traceId = {} \", traceId);\n\n return userService.list();\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"測試 ","attrs":{}},{"type":"link","attrs":{"href":"http://localhost:8000/user/list","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"http://localhost:8000/user/list","attrs":{}}]}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/12/126da14c48930793186de5f355210f7f.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在Skywalking UI中查詢tranceId","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d3/d3fea9bb9341090603de2a067876f798.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.4.1 @Trace將方法加入追蹤鏈路","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果一個業務方法想在ui界面的跟蹤鏈路上顯示出來,只需要在業務方法上加上@Trace註解即可","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9b/9bfc699a91479221acf1f9304e8f08ca.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"測試:","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/56/565e5ad11cb8287886409728cdbebb82.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.4.2 加入@Tags或@Tag","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們還可以爲追蹤鏈路增加其他額外的信息,比如記錄參數和返回信息。實現方式:在方法上增加@Tag或者@Tags。","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"@Trace\n@Tag(key = \"list\", value = \"returnedObj\")\npublic List list(){\n return userMapper.list();\n}\n\n@Trace\n@Tags({@Tag(key = \"param\", value = \"arg[0]\"),\n @Tag(key = \"user\", value = \"returnedObj\")})\npublic User getById(Integer id){\n return userMapper.getById(id);\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/59/59b90c918eab5863c3b7e88f5bc4a6ad.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/88/88b08a9bfeae4c2bcee5446e858f8284.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2.5 Skywalking集成日誌框架","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://github.com/apache/skywalking/blob/master/docs/en/setup/service-agent/java-agent/Application-toolkit-logback-1.x.md","title":null,"type":null},"content":[{"type":"text","text":"logback官方配置","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://skywalking.apache.org/docs/main/v8.4.0/en/setup/service-agent/java-agent/application-toolkit-log4j-1.x/","title":null,"type":null},"content":[{"type":"text","text":"log4j官方配置","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://skywalking.apache.org/docs/main/v8.4.0/en/setup/service-agent/java-agent/application-toolkit-log4j-2.x","title":null,"type":null},"content":[{"type":"text","text":"log4j2j官方配置","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"引入依賴","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\n\n org.apache.skywalking\n apm-toolkit-logback-1.x\n 8.4.0\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"添加logback-spring.xml文件,並配置 %tid 佔位符","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b6/b63d88c3f42b3954cb01dc3d44492355.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"測試","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a8/a880d10995ed435cafb31b45f2c347aa.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Skywalking通過grpc上報日誌 (需要v8.4.0)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"gRPC報告程序可以將收集到的日誌轉發到SkyWalking OAP服務器上","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"logback-spring.xml中添加","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":" \n \n\n \n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/70/7010720eaafc75b0ec7fe0400374c121.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"打開agent/config/agent.config配置文件,添加如下配置信息:","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"plugin.toolkit.log.grpc.reporter.server_host=${SW_GRPC_LOG_SERVER_HOST:192.168.3.100}\nplugin.toolkit.log.grpc.reporter.server_port=${SW_GRPC_LOG_SERVER_PORT:11800}\nplugin.toolkit.log.grpc.reporter.max_message_size=${SW_GRPC_LOG_MAX_MESSAGE_SIZE:10485760}\nplugin.toolkit.log.grpc.reporter.upstream_timeout=${SW_GRPC_LOG_GRPC_UPSTREAM_TIMEOUT:30}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上配置是默認配置信息,agent與oap在本地的可以不配","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/89/89968428f4155f33e57e44d13552f1d0.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://skywalking.apache.org/docs/main/v8.4.0/en/setup/service-agent/java-agent/readme/#table-of-agent-configuration-properties","title":null,"type":null},"content":[{"type":"text","text":"agent配置信息大全","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Skywalking UI效果","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d6/d690d4379e9350cfd4fe3f4a7941585e.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此處日期格式存在問題","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://github.com/apache/skywalking-rocketbot-ui/pull/428","title":null,"type":null},"content":[{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"https://github.com/apache/skywalking-rocketbot-ui/pull/428","attrs":{}}]}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e0/e073a5a1e9d4d3c3874d6b7cc95082af.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2.6 Skywalking集羣部署","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Skywalking集羣是將skywalking oap作爲一個服務註冊到nacos上,只要skywalking oap服務沒有全部宕機,保證有一個skywalking oap在運行,就能進行跟蹤。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"搭建一個skywalking oap集羣需要:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"(1)至少一個Nacos(也可以是nacos集羣)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"(2)至少一個ElasticSearch(也可以是es集羣)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"(3)至少2個skywalking oap服務;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"(4)至少1個UI(UI也可以集羣多個,用Nginx代理統一入口)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.修改config/application.yml文件","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用nacos作爲註冊中心","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/60/60ed9457f630d87ff6c5eb1d6bbaba74.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"修改nacos配置","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/08/0881630a758cefd6cd761cf8bdef15a1.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以選擇性修改監聽端口","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/08/08504e7f00f48677430aacbac2f3d49d.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"修改存儲策略,使用elasticsearch7作爲storage","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/98/985ac9cc71a07f64c8dfd2ca874bb482.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/75/75623d351f6a4d4280c60de3b6a6a1cf.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2. 配置ui服務webapp.yml文件的listOfServers,寫兩個地址","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ba/ba9f4900718ccc3bd764d74281785149.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3.啓動服務測試","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"啓動Skywalking服務,指定springboot應用的jvm參數","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"-DSW_AGENT_COLLECTOR_BACKEND_SERVICES=192.168.3.10:11800,192.168.3.12:11800","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章