{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"用戶將軟件從 x86 平臺移植到華爲鯤鵬平臺時，由於平臺間體系結構和指令集各不相同，這時就需要利用跨平臺工具鏈對源代碼進行重新構建，這中間就會碰到一些棘手的問題，比如編譯器無法直接跨平臺編譯源代碼工程中平臺相關的彙編代碼，又或者當用戶把應用遷移到鯤鵬平臺上之後，鯤鵬多核的優勢卻無法充分發揮。因此，如何把 x86 源代碼中的彙編代碼高效識別並遷移到鯤鵬平臺上，同時保證遷移後的性能，是鯤鵬生態建設和拓展過程中急需解決的一個技術課題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"InfoQ 記者有幸在"},{"type":"link","attrs":{"href":"https:\/\/qcon.infoq.cn\/2021\/beijing\/","title":"xxx","type":null},"content":[{"type":"text","text":"QCon 2021 全球軟件開發大會"}]},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"上，就這個問題採訪到了華爲鯤鵬編譯技術專家樂永年，他的主要研究方向是鯤鵬平臺的編譯技術和應用性能優化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"以下是視頻採訪的全部內容，爲方便讀者查看，視頻下方也附上了文字內容。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"embedcomp","attrs":{"type":"video","data":{"id":"385377","name":"華爲鯤鵬編譯技術專家樂永年","poster":"https:\/\/static001.infoq.cn\/resource\/image\/d3\/8b\/d36f986cec566ab47f438e133fa74e8b.png","url":"https:\/\/media001.geekbang.org\/0ec2886027c44b1aa5685d9c006307ad\/d9042efc66d0445aa2b760af9bf9d3af-579b02c47b66a55a30ac5b497003950f-sd.m3u8"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ：樂老師您好，歡迎來到QCon並接受我們的採訪，請您先自我介紹一下吧。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"樂老師：好的，我的名字叫樂永年，加入華爲之前我從事了20多年的軟件開發工作，包括編譯器、加速庫以及軟件的性能調優我都幹過，目前我在華爲鯤鵬擔任編譯器專家，從事鯤鵬開發套件相關的設計和架構工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ：您選擇加入華爲鯤鵬主要有哪些方面的考慮？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"樂老師：剛纔也提到了，我之前從事了相當多的軟件開發工作。最近鯤鵬正在下一盤大棋，它實際上從硬件開放、從基礎軟件的開源、到面向生態開發者的"},{"type":"link","attrs":{"href":"https:\/\/bbs.huaweicloud.com\/live\/kunpeng_live\/202103271600.html","title":"xxx","type":null},"content":[{"type":"text","text":"DevKit開發套件"}]},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"以及"},{"type":"link","attrs":{"href":"https:\/\/bbs.huaweicloud.com\/live\/kunpeng_live\/202104151930.html","title":"xxx","type":null},"content":[{"type":"text","text":"鯤鵬應用使能套件BoostKit"}]},{"type":"text","text":"，全系列全棧投入，打造一個新的計算底座以及新的計算生態，讓我覺得這裏面有很多機會。我之前一直有一個夢想，我希望我做的產品能夠爲廣大用戶所使用，同時做的工作能夠改變大衆的生活，這樣看的話鯤鵬對我來說是個理想的工作，能幫助我實現夢想，我也希望在鯤鵬這一塊繼續努力添磚加瓦。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ：那您現在在鯤鵬做的事情主要有哪些技術挑戰？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"樂老師：從大的角度來說，計算主要是通過算力來衡量，算力是數據處理的能力，它廣泛存在於各個硬件設備裏面，包括我們的手機，包括我們的PC，包括我們的硬件服務器。從我們各領域角度來看，我們的數據中心，以及我們的高鐵，行行業業方方面面都需要算力的支撐。但是隨着最近大數據以及AI的蓬勃發展，原有的算力已經跟不上現有發展的需要，人民日益增長的物質文化需求滿足不了了，所以我們需要從各個方面考慮，比如說我們的硬件，包括我們的硬件開放、軟件開源，擰在一起，和我們的生態夥伴一起形成合力，打造新的生態，打造新的算力。這裏有很多挑戰，比如傳統的摩爾定律，已經不能滿足發展需要，原來的單一架構也不能滿足算力的增長需要，這裏我們要和生態同仁一起把這個新的生態、新的算力打造好。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ：您在會議上演講主題是關於跨平臺遷移的，那麼從其他平臺到華爲鯤鵬的跨平臺遷移過程中，有哪些技術難點？你們是怎麼解決的？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"樂老師：我們打造了DevKit這個面向開發者的開發套件，它提供了一站式全流程的工作方式，包含了遷移，包含了調優，也包含了編譯以及診斷等一系列能力。在這裏它主要解決兩方面的問題。第一個是把應用從X86平臺遷移到鯤鵬平臺，同時保持遷移以後的性能，因此遷移在這裏佔了一個主要的地位。在這個遷移過程中有兩套方案，針對不同的場景。一個就是有源碼遷移，一個就是無源碼遷移。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"對於有源碼遷移，這裏碰到的挑戰不少。首先，應用的語言多種多樣。其次，編譯的選項很複雜，很多編譯的選項跟平臺密切綁定，密切相關。第三個點就是在構建腳本里，很多構建腳本是動態執行的，這些都需要在開發套件裏，通過代碼遷移工具插件得到一個很好的解決。比如說這裏的代碼遷移工具插件能夠提供八種語言的支持，同時這裏需要提供一個靜態的掃描，把用戶和平臺相關的一些編譯選項提示出來，讓用戶修改。我們也在攻關的動態執行技術，能夠讓構建腳本在靜態的方式下得到解釋執行，把平臺相關的問題都找到，這些是有源碼遷移的難點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"對於無源碼遷移也有不少挑戰，比如說我們怎麼保證遷移後的應用性能衰減儘量少，這裏動態二進制的優化技術可以幫助我們把這個問題克服。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ：華爲鯤鵬原生編譯器畢昇是你們的產品，在設計這個編譯器的過程中，有哪些有意思的技術難點？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"樂老師：畢昇編譯器是我們華爲鯤鵬原生的編譯器，它通過挖掘鯤鵬架構特點，提供最佳的代碼質量，發揮鯤鵬原生性能優勢，這裏競爭力構築的方向就是高性能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"對於服務器而言大概有這幾個模塊，第一塊是CPU，第二塊是內存，第三塊是存儲I\/O，第四塊網絡I\/O，這四塊是密切相關的，對性能最有影響。編譯器主要影響第一和第二這兩個部件，一個是CPU，一個是內存，圍繞這兩個部件，我們絞盡腦汁想各種各樣優化的手段，我們這裏的畢昇編譯器的架構師"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}}],"text":"魏偉爲此"},{"type":"text","text":"頭髮也慢慢變白了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"我這裏具體展開講一講，對於CPU而言這裏是比較傳統的一種優化，對於鯤鵬特有的流水線架構，這裏進行了指令調度優化。對於鯤鵬特有的NEON指令集，我們也開發了不少並行化和矢量化的優化技術，把NEON指令的能力給充分利用起來，這是傳統優化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"對於內存而言，它實際上越來越成爲性能的瓶頸，因爲CPU的速度越來越快，但是內存的訪存速度還是趕不上CPU發展的速度，我們在這裏也開發了很多的優化手段，比如說我們對數組進行重排，同時對結構體進行佈局的優化，通過一系列內存排布的優化大幅提升應用性能，除此之外我們還對鯤鵬本身的架構有針對性地對內存"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}}],"text":"緩存的預取做"},{"type":"text","text":"了調整和增強，通過這些手段從CPU和內存的角度發揮硬件本身的能力，大幅提升了軟件的性能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ：剛纔我們一直在談性能的問題，我想深入瞭解一下，你們是怎麼保證程序移植後的性能問題呢？在性能保障這方面有哪些措施？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"樂老師：鯤鵬DevKit主要的工作，或者業務基本面，就是保證應用遷移，其次是保證應用遷移以後的性能得到維持，現在你這個問題就是關於性能維持方面。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"性能維持肯定不能只是靠一兩個手段，我們從全流程開發套件使用過程中，在每個階段都要考慮性能，在代碼遷移階段，我們有加速庫插件，它能夠針對代碼裏面常用的熱點函數，進行自動識別和智能聯想，做到鯤鵬親和加速庫的函數的一鍵替換，這是第一個。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"其次是在編譯階段，我們這裏還有鯤鵬原生的畢昇編譯器，能夠挖掘架構本身特點，把我們應用的性能潛力充分發揮出來，這是第二個。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"第三個是我們還有一個獨特的專門針對性能的調優工具，叫做性能分析工具。這個性能分析工具能夠提供場景化和全系統的分析能力。幫助用戶查找應用中性能的瓶頸，提供專業化的性能調優的建議，這是第三點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"第四點是我們還打造了一個新的工具，這個工具就是異常診斷工具，這個異常診斷工具能夠幫助用戶診斷常見的性能異常，同時快速定位性能問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"通過這四種工具在源碼級別的優化，保證遷移以後的性能能夠維持。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"對於二進制代碼沒有源碼的情況，我們這裏還提供了動態二進制的翻譯工具，這個翻譯工具裏面包含了大量的動態優化，能夠把遷移過程中的的性能損耗降到最小，保證應用遷移後性能得到維持。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ：除了快速遷移彙編代碼的技術，DevKit在設計過程中還解決了哪些關鍵難題？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"樂老師：我們還有無源碼遷移的動態二進制翻譯技術。實際上這兩個問題是有點類似的，對於有源碼的應用而言，有源碼的代碼裏它有彙編，需要通過指令的翻譯把原有的彙編代碼變成在鯤鵬上的彙編代碼，是種語言到語言的翻譯。這是彙編代碼範疇，和它對應的是指令。一個二進制應用程序，它沒有源代碼了，這時候我怎麼在鯤鵬平臺上很好地執行這個應用，這裏用了一個類似的技術，叫動態二進制翻譯技術，這個動態二進制翻譯技術就是把一條條指令拆解出來，去動態地解釋和執行，當然這個解釋執行過程中性能肯定會有損耗，而且損耗還不少，所以在這個過程中會有大量的動態二進制優化技術在裏面起作用，把性能損耗降到最少，保證遷移以後用戶還可以使用這個應用。到目前爲止，我們能夠把性能損耗控制在15%到20%以內。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ：DevKit在哪些領域和行業有成功的應用呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"樂老師：DevKit實際上跟鯤鵬平臺密切相關，我們的鯤鵬平臺在政府、金融、交通、電力各個領域都有非常不錯的進展。在這裏DevKit作爲面向開發者使"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}}],"text":"能的"},{"type":"text","text":"重要抓手，對於整個鯤鵬平臺的拓展起了一個非常大的促進作用。到目前爲止，我們鯤鵬的DevKit，已經有五萬開發用戶，有三百多個合作伙伴，和我們一起打造鯤鵬的算力新生態。同時，這裏打個廣告，我們還有鯤鵬應用創新大賽，所有的參賽團隊都會使用鯤鵬開發套件，打造他們的參賽作品，進行應用遷移、開發、調優和調試。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"InfoQ：好的，謝謝樂老師。"}]}]}
鯤鵬DevKit：鯤鵬平臺代碼遷移利器｜QCon
{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"用戶將軟件從 x86 平臺移植到華爲鯤鵬平臺時，由於平臺間體系結構和指令集各不相同，這時就需要利用跨平臺工具鏈對源代碼進行重新構建，這中間就會碰到一些棘手的問題，比如編譯器無法直接跨平臺編譯源代碼工程中平臺相關的彙編代碼，又或者當用戶把應用遷移到鯤鵬平臺上之後，鯤鵬多核的優勢卻無法充分發揮。因此，如何把 x86 源代碼中的彙編代碼高效識別並遷移到鯤鵬平臺上，同時保證遷移後的性能，是鯤鵬生態建設和拓展過程中急需解決的一個技術課題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"InfoQ 記者有幸在"},{"type":"link","attrs":{"href":"https:\/\/qcon.infoq.cn\/2021\/beijing\/","title":"xxx","type":null},"content":[{"type":"text","text":"QCon 2021 全球軟件開發大會"}]},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"上，就這個問題採訪到了華爲鯤鵬編譯技術專家樂永年，他的主要研究方向是鯤鵬平臺的編譯技術和應用性能優化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"以下是視頻採訪的全部內容，爲方便讀者查看，視頻下方也附上了文字內容。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"embedcomp","attrs":{"type":"video","data":{"id":"385377","name":"華爲鯤鵬編譯技術專家樂永年","poster":"https:\/\/static001.infoq.cn\/resource\/image\/d3\/8b\/d36f986cec566ab47f438e133fa74e8b.png","url":"https:\/\/media001.geekbang.org\/0ec2886027c44b1aa5685d9c006307ad\/d9042efc66d0445aa2b760af9bf9d3af-579b02c47b66a55a30ac5b497003950f-sd.m3u8"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ：樂老師您好，歡迎來到QCon並接受我們的採訪，請您先自我介紹一下吧。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"樂老師：好的，我的名字叫樂永年，加入華爲之前我從事了20多年的軟件開發工作，包括編譯器、加速庫以及軟件的性能調優我都幹過，目前我在華爲鯤鵬擔任編譯器專家，從事鯤鵬開發套件相關的設計和架構工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ：您選擇加入華爲鯤鵬主要有哪些方面的考慮？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"樂老師：剛纔也提到了，我之前從事了相當多的軟件開發工作。最近鯤鵬正在下一盤大棋，它實際上從硬件開放、從基礎軟件的開源、到面向生態開發者的"},{"type":"link","attrs":{"href":"https:\/\/bbs.huaweicloud.com\/live\/kunpeng_live\/202103271600.html","title":"xxx","type":null},"content":[{"type":"text","text":"DevKit開發套件"}]},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"以及"},{"type":"link","attrs":{"href":"https:\/\/bbs.huaweicloud.com\/live\/kunpeng_live\/202104151930.html","title":"xxx","type":null},"content":[{"type":"text","text":"鯤鵬應用使能套件BoostKit"}]},{"type":"text","text":"，全系列全棧投入，打造一個新的計算底座以及新的計算生態，讓我覺得這裏面有很多機會。我之前一直有一個夢想，我希望我做的產品能夠爲廣大用戶所使用，同時做的工作能夠改變大衆的生活，這樣看的話鯤鵬對我來說是個理想的工作，能幫助我實現夢想，我也希望在鯤鵬這一塊繼續努力添磚加瓦。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ：那您現在在鯤鵬做的事情主要有哪些技術挑戰？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"樂老師：從大的角度來說，計算主要是通過算力來衡量，算力是數據處理的能力，它廣泛存在於各個硬件設備裏面，包括我們的手機，包括我們的PC，包括我們的硬件服務器。從我們各領域角度來看，我們的數據中心，以及我們的高鐵，行行業業方方面面都需要算力的支撐。但是隨着最近大數據以及AI的蓬勃發展，原有的算力已經跟不上現有發展的需要，人民日益增長的物質文化需求滿足不了了，所以我們需要從各個方面考慮，比如說我們的硬件，包括我們的硬件開放、軟件開源，擰在一起，和我們的生態夥伴一起形成合力，打造新的生態，打造新的算力。這裏有很多挑戰，比如傳統的摩爾定律，已經不能滿足發展需要，原來的單一架構也不能滿足算力的增長需要，這裏我們要和生態同仁一起把這個新的生態、新的算力打造好。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ：您在會議上演講主題是關於跨平臺遷移的，那麼從其他平臺到華爲鯤鵬的跨平臺遷移過程中，有哪些技術難點？你們是怎麼解決的？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"樂老師：我們打造了DevKit這個面向開發者的開發套件，它提供了一站式全流程的工作方式，包含了遷移，包含了調優，也包含了編譯以及診斷等一系列能力。在這裏它主要解決兩方面的問題。第一個是把應用從X86平臺遷移到鯤鵬平臺，同時保持遷移以後的性能，因此遷移在這裏佔了一個主要的地位。在這個遷移過程中有兩套方案，針對不同的場景。一個就是有源碼遷移，一個就是無源碼遷移。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"對於有源碼遷移，這裏碰到的挑戰不少。首先，應用的語言多種多樣。其次，編譯的選項很複雜，很多編譯的選項跟平臺密切綁定，密切相關。第三個點就是在構建腳本里，很多構建腳本是動態執行的，這些都需要在開發套件裏，通過代碼遷移工具插件得到一個很好的解決。比如說這裏的代碼遷移工具插件能夠提供八種語言的支持，同時這裏需要提供一個靜態的掃描，把用戶和平臺相關的一些編譯選項提示出來，讓用戶修改。我們也在攻關的動態執行技術，能夠讓構建腳本在靜態的方式下得到解釋執行，把平臺相關的問題都找到，這些是有源碼遷移的難點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"對於無源碼遷移也有不少挑戰，比如說我們怎麼保證遷移後的應用性能衰減儘量少，這裏動態二進制的優化技術可以幫助我們把這個問題克服。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ：華爲鯤鵬原生編譯器畢昇是你們的產品，在設計這個編譯器的過程中，有哪些有意思的技術難點？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"樂老師：畢昇編譯器是我們華爲鯤鵬原生的編譯器，它通過挖掘鯤鵬架構特點，提供最佳的代碼質量，發揮鯤鵬原生性能優勢，這裏競爭力構築的方向就是高性能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"對於服務器而言大概有這幾個模塊，第一塊是CPU，第二塊是內存，第三塊是存儲I\/O，第四塊網絡I\/O，這四塊是密切相關的，對性能最有影響。編譯器主要影響第一和第二這兩個部件，一個是CPU，一個是內存，圍繞這兩個部件，我們絞盡腦汁想各種各樣優化的手段，我們這裏的畢昇編譯器的架構師"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}}],"text":"魏偉爲此"},{"type":"text","text":"頭髮也慢慢變白了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"我這裏具體展開講一講，對於CPU而言這裏是比較傳統的一種優化，對於鯤鵬特有的流水線架構，這裏進行了指令調度優化。對於鯤鵬特有的NEON指令集，我們也開發了不少並行化和矢量化的優化技術，把NEON指令的能力給充分利用起來，這是傳統優化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"對於內存而言，它實際上越來越成爲性能的瓶頸，因爲CPU的速度越來越快，但是內存的訪存速度還是趕不上CPU發展的速度，我們在這裏也開發了很多的優化手段，比如說我們對數組進行重排，同時對結構體進行佈局的優化，通過一系列內存排布的優化大幅提升應用性能，除此之外我們還對鯤鵬本身的架構有針對性地對內存"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}}],"text":"緩存的預取做"},{"type":"text","text":"了調整和增強，通過這些手段從CPU和內存的角度發揮硬件本身的能力，大幅提升了軟件的性能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ：剛纔我們一直在談性能的問題，我想深入瞭解一下，你們是怎麼保證程序移植後的性能問題呢？在性能保障這方面有哪些措施？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"樂老師：鯤鵬DevKit主要的工作，或者業務基本面，就是保證應用遷移，其次是保證應用遷移以後的性能得到維持，現在你這個問題就是關於性能維持方面。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"性能維持肯定不能只是靠一兩個手段，我們從全流程開發套件使用過程中，在每個階段都要考慮性能，在代碼遷移階段，我們有加速庫插件，它能夠針對代碼裏面常用的熱點函數，進行自動識別和智能聯想，做到鯤鵬親和加速庫的函數的一鍵替換，這是第一個。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"其次是在編譯階段，我們這裏還有鯤鵬原生的畢昇編譯器，能夠挖掘架構本身特點，把我們應用的性能潛力充分發揮出來，這是第二個。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"第三個是我們還有一個獨特的專門針對性能的調優工具，叫做性能分析工具。這個性能分析工具能夠提供場景化和全系統的分析能力。幫助用戶查找應用中性能的瓶頸，提供專業化的性能調優的建議，這是第三點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"第四點是我們還打造了一個新的工具，這個工具就是異常診斷工具，這個異常診斷工具能夠幫助用戶診斷常見的性能異常，同時快速定位性能問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"通過這四種工具在源碼級別的優化，保證遷移以後的性能能夠維持。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"對於二進制代碼沒有源碼的情況，我們這裏還提供了動態二進制的翻譯工具，這個翻譯工具裏面包含了大量的動態優化，能夠把遷移過程中的的性能損耗降到最小，保證應用遷移後性能得到維持。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ：除了快速遷移彙編代碼的技術，DevKit在設計過程中還解決了哪些關鍵難題？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"樂老師：我們還有無源碼遷移的動態二進制翻譯技術。實際上這兩個問題是有點類似的，對於有源碼的應用而言，有源碼的代碼裏它有彙編，需要通過指令的翻譯把原有的彙編代碼變成在鯤鵬上的彙編代碼，是種語言到語言的翻譯。這是彙編代碼範疇，和它對應的是指令。一個二進制應用程序，它沒有源代碼了，這時候我怎麼在鯤鵬平臺上很好地執行這個應用，這裏用了一個類似的技術，叫動態二進制翻譯技術，這個動態二進制翻譯技術就是把一條條指令拆解出來，去動態地解釋和執行，當然這個解釋執行過程中性能肯定會有損耗，而且損耗還不少，所以在這個過程中會有大量的動態二進制優化技術在裏面起作用，把性能損耗降到最少，保證遷移以後用戶還可以使用這個應用。到目前爲止，我們能夠把性能損耗控制在15%到20%以內。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"InfoQ：DevKit在哪些領域和行業有成功的應用呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"樂老師：DevKit實際上跟鯤鵬平臺密切相關，我們的鯤鵬平臺在政府、金融、交通、電力各個領域都有非常不錯的進展。在這裏DevKit作爲面向開發者使"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}}],"text":"能的"},{"type":"text","text":"重要抓手，對於整個鯤鵬平臺的拓展起了一個非常大的促進作用。到目前爲止，我們鯤鵬的DevKit，已經有五萬開發用戶，有三百多個合作伙伴，和我們一起打造鯤鵬的算力新生態。同時，這裏打個廣告，我們還有鯤鵬應用創新大賽，所有的參賽團隊都會使用鯤鵬開發套件，打造他們的參賽作品，進行應用遷移、開發、調優和調試。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"InfoQ：好的，謝謝樂老師。"}]}]}
發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章