寫在前面：我是【程序員寶藏】的寶藏派發員，致力於創作原創乾貨。我熱愛技術、熱愛開源與分享，創作的【計算機基礎面試問題】系列文章和【計算機基礎主幹知識】系列文章廣受好評！後期會創作更多優質原創系列文章！如果您對計算機基礎知識、編程等感興趣，可以關注我，我們一起成長！
本人力薦：如果覺得CSDN排版不夠美觀，歡迎來我的個人原創公zong號【程序員寶藏】（號如其名，誠不欺你！）查看有紅色重點標記和排版美觀的全系列文章（不細你來找我要紅包）
參考推文鏈接：TCP三次握手四次揮手
好多同學問我要pdf版，我乾脆把我的全部原創文章都整理成了pdf直接打印版，在公zong號後臺回覆關鍵字【寶藏】即可免費帶回家慢慢看！
本系列參考文章：
文章目錄
三、大數據相關
1.基於高性能密碼實現的大數據安全方案
A Big Data Security Scheme Based on High-Performance Cryptography Implementation
摘要： 目前信息技術發展的趨勢是以大數據計算爲基礎的人工智能技術.雲計算、霧計算、邊緣計算等計算模式下的大數據處理技術，在給經濟發展帶來巨大推動力的同時，也面臨着巨大的安全風險.密碼技術是解決大數據安全的核心技術.大數據的機密性、認證性及隱私保護問題需要解決海量數據的高速加解密問題；高併發的大規模用戶認證問題；大數據的隱私保護及密態計算問題等，這些問題的解決，需要底層密碼算法的快速實現.針對大數據安全應用的邏輯架構，對底層的國產密碼標準算法SM4-XTS，SM2以及大整數模冪運算，分別給出快速計算的算法，並在基於Xilinx公司的KC705開發板上進行了驗證，並給出實驗數據.實驗表明：該工作具有一定的先進性：1)SM4-XTS模式的實現填補了國內該方向的空白；2)SM2簽名具有較高性能，領先於國內同類產品；3)大整數的模冪運算應用於同態密碼的產品化，填補了國內該產品的空白.
關鍵詞: SM4-XTS, SM2, 大整數模冪, 密碼算法快速實現, 大數據
Abstract: At present, the trend of information technology development is the artificial intelligence technology based on big data computing. Although it has made enormous contribution in the economic development, big data processing technology which includes cloud computing, fog computing, edge computing and other computing modes also brings a great risk of data security. Cryptographic technology is the kernel of the big data security. Confidentiality, authentication and privacy protection of big data need to solve the following three security problems: firstly, high-speed encryption and decryption of massive data; secondly, the authentication problem of high concurrency and large scale user; thirdly, privacy protection in data mining. The solution of these problems requires the fast implementation of the underlying cryptographic algorithm. Aiming at the logic architecture of big data security application, this paper gives a fast calculation algorithm for the cryptographic standard algorithm SM4-XTS, SM2 and modular exponentiation of large integers. It is verified on the KC705 development board based on Xilinx company, the results of experiment show that our work has certain advancement: 1) The implementation of SM4-XTS fills the blank of this direction in China. 2) SM2 signature has high performance, leading domestic similar products. 3) Modular exponentiation is applied to the productization of homomorphism cryptography, and its performance is ahead of other similar products.
Key words: SM4-XTS, SM2, modular exponentiation, high-speed implementation of cryptographic algorithm, big data
2.知識圖譜研究綜述及其在醫療領域的應用。
Research Review of Knowledge Graph and Its Application in Medical Domain
摘要： 隨着醫療大數據時代的到來，知識互聯受到了廣泛的關注.如何從海量的數據中提取有用的醫學知識，是醫療大數據分析的關鍵.知識圖譜技術提供了一種從海量文本和圖像中抽取結構化知識的手段，知識圖譜與大數據技術、深度學習技術相結合，正在成爲推動人工智能發展的核心驅動力.知識圖譜技術在醫療領域擁有廣闊的應用前景，該技術在醫療領域的應用研究將會在解決優質醫療資源供給不足和醫療服務需求持續增加的矛盾中產生重要的作用.目前，針對醫學知識圖譜的研究還處於探索階段，現有知識圖譜技術在醫療領域普遍存在效率低、限制多、拓展性差等問題.首先針對醫療領域大數據專業性強、結構複雜等特點，對醫學知識圖譜架構和構建技術進行了全面剖析;其次，分別針對醫學知識圖譜中知識表示、知識抽取、知識融合和知識推理這4個模塊的關鍵技術和研究進展進行綜述，並對這些技術進行實驗分析與比較.此外，介紹了醫學知識圖譜在臨牀決策支持、醫療智能語義檢索、醫療問答等醫療服務中的應用現狀.最後對當前研究存在的問題與挑戰進行了討論和分析，並對其發展前景進行了展望.
關鍵詞: 知識圖譜, 智慧醫療, 大數據, 知識融合, 自然語言處理
Abstract: With the advent of the medical big data era, knowledge interconnection has received extensive attention. How to extract useful medical knowledge from massive data is the key for medical big data analysis. Knowledge graph technology provides a means to extract structured knowledge from massive texts and images.The combination of knowledge graph, big data technology and deep learning technology is becoming the core driving force for the development of artificial intelligence. The knowledge graph technology has a broad application prospect in the medical domain. The application of knowledge graph technology in the medical domain will play an important role in solving the contradiction between the supply of high-quality medical resources and the continuous increase of demand for medical services.At present, the research on medical knowledge graph is still in the exploratory stage. The existing knowledge graph technology generally has several problems such as low efficiency, multiple restrictions and poor expansion in the medical domain. This paper firstly analyzes the medical knowledge graph architecture and construction technology for the strong professionalism and complex structure of big data in the medical domain. Secondly, the key technologies and research progress of the three modules of knowledge extraction, knowledge expression, knowledge fusion and knowledge reasoning in medical knowledge map are summarized. In addition, the application status of medical knowledge maps in clinical decision support, medical intelligence semantic retrieval, medical question answering system and other medical services are introduced. Finally, the existing problems and challenges of current research are discussed and analyzed, and its development is prospected.
Key words: knowledge graph, medical wisdom, big data, knowledge fusion, natural language processing
3.基於MOOC數據的學習行爲分析與預測
Learning Behavior Analysis and Prediction Based on MOOC Data
摘要： 隨着近2年慕課(massive open online course, MOOC)的興起，教育大數據分析正成爲一個新興的研究方向.2013年秋，北京大學在Coursera上開設了6門慕課.通過分析挖掘約8萬多人次參與這6門課的海量學習行爲數據，力圖展現慕課學習活動多個側面的風貌.同時，首次針對中文慕課中學習行爲的特點，將學習者分類，以更加深入地考察學習行爲與學習效果之間的關係.在此基礎上，通過選擇學習者的若干典型行爲特徵，對他們最後的學習成果進行預測的工作也尚屬首次.數據表明：基於學習行爲的特徵分析能有效地判別一個學習者能否成功完成學習任務獲得通過證書，並能找出潛在的認真學習者，這爲今後更加精準的慕課教學測評提供了一種依據.
關鍵詞: 慕課, 學習者類型, 學習行爲, 數據分析, 成績預測
Abstract: With the booming of MOOC (massive open online course) in the past two years, educational data analysis has become a promising research field where the quality of teaching and learning can be and is being quantified to improve the educational effectiveness and even to promote the modern higher education. In the autumn of 2013, Peking University released its first six courses on the Coursera platform. Through mining and analyzing the massive data of learning behavior of over 80000 participants from the courses, this paper endeavors to manifest more than one side of learning activity in MOOC. Meanwhile, according to the characteristic of learning behavior in Chinese MOOC, learners are classified into several groups and then the relationship between their learning behavior and performance is thoroughly studied. Based on the above work, we find out that learners performance, regarding whether heshe could get certificated eventually, can be predicted by looking into several features of their learning behavior. Experiment results indicate that these features can be trained to effectively estimate whether a learner is probably to complete the course successfully. Besides, this method has the potential to partially evaluate the quality of both teaching and learning in practice.
Key words: massive open online course (MOOC), engagement style, learning behavior, data analysis, performance prediction
4.知識圖譜構建技術綜述
Knowledge Graph Construction Techniques
摘要： 谷歌知識圖譜技術近年來引起了廣泛關注，由於公開披露的技術資料較少，使人一時難以看清該技術的內涵和價值.從知識圖譜的定義和技術架構出發，對構建知識圖譜涉及的關鍵技術進行了自底向上的全面解析.1)對知識圖譜的定義和內涵進行了說明，並給出了構建知識圖譜的技術框架，按照輸入的知識素材的抽象程度將其劃分爲3個層次：信息抽取層、知識融合層和知識加工層;2)分別對每個層次涉及的關鍵技術的研究現狀進行分類說明，逐步揭示知識圖譜技術的奧祕，及其與相關學科領域的關係;3)對知識圖譜構建技術當前面臨的重大挑戰和關鍵問題進行了總結.
關鍵詞: 知識圖譜, 語義網, 信息檢索, 語義搜索引擎, 自然語言處理
Abstract: Google’s knowledge graph technology has drawn a lot of research attentions in recent years. However, due to the limited public disclosure of technical details, people find it difficult to understand the connotation and value of this technology. In this paper, we introduce the key techniques involved in the construction of knowledge graph in a bottom-up way, starting from a clearly defined concept and a technical architecture of the knowledge graph. Firstly, we describe in detail the definition and connotation of the knowledge graph, and then we propose the technical framework for knowledge graph construction, in which the construction process is divided into three levels according to the abstract level of the input knowledge materials, including the information extraction layer, the knowledge integration layer, and the knowledge processing layer, respectively. Secondly, the research status of the key technologies for each level are surveyed comprehensively and also investigated critically for the purposes of gradually revealing the mysteries of the knowledge graph technology, the state-of-the-art progress, and its relationship with related disciplines. Finally, five major research challenges in this area are summarized, and the corresponding key research issues are highlighted.
Key words: knowledge graph, semantic Web, information retrieval, semantic search engine, natural language processing
5.一種面向大規模序列數據的交互特徵並行挖掘算法
A Parallel Algorithm for Mining Interactive Features from Large Scale Sequences
摘要： 序列是一種重要的數據類型，在諸多應用領域廣泛存在.基於序列的特徵選擇具有廣闊的現實應用場景.交互特徵是指一組整體具有顯著強於單獨個體與目標相關性的特徵集合.從大規模序列中挖掘交互特徵面臨着位點的“組合爆炸”問題，計算挑戰性極大.針對該問題，以生物領域高通量測序數據爲背景，提出了一種新的基於並行處理和演化計算的高階交互特徵挖掘算法.位點數是制約交互作用挖掘效率的根本因素.擯棄了現有方法基於序列分塊的並行策略，採用基於位點分塊的並行思想，具有天然的效率優勢.進一步，提出了極大等位公共子序列(maximal allelic common subsequence, MACS)的概念並設計了基於MACS的特徵區域劃分策略.該策略能將交互特徵的查找範圍縮小至許多“碎片”空間，並保證不同“碎片”間不存在交互特徵，避免計算耦合引起的高額通信代價.利用基於置換搜索的並行蟻羣算法，執行交互特徵選擇.大量真實數據集和合成數據集上的實驗結果，證實提出的PACOIFS算法在有效性和效率上優於同類其他算法.
關鍵詞: 交互特徵, 數據挖掘, 大規模序列, 蟻羣算法, 並行計算, 極大等位公共子序列
Abstract: Sequence is an important type of data which is widely existing in various domains, and thus feature selection from sequence data is of practical significance in extensive applications. Interactive features refer to a set of features, each of which is weakly correlated with the target, but the whole of which is strongly correlated with the target. It is of great challenge to mine interactive features from large scale sequence data for the combinatorial explosion problem of loci. To address the problem, against the background of high-throughput sequencing in biology, a parallel evolutionary algorithm for high-order interactive features mining is proposed in this paper. Instead of sequence-block based parallel strategy, the work is inspired by loci-based idea since the number of loci is the fundamental factor that restricts the efficiency. Further, we propose the conception of maximal allelic common subsequence (MACS) and MACS based strategy for feature region partition. According to the strategy, the search range of interactive features is narrowed to many fragged spaces and interactions are guaranteed not to exist among different fragments. Finally, a parallel ant algorithm based on substitution search is developed to conduct interactive feature selection. Extensive experiments on real and synthetic datasets show that the efficiency and effectiveness of the proposed PACOIFS algorithm is superior to that of competitive algorithms.
Key words: interactive features, data mining, large scale sequence, ant colony algorithm, parallel computation, maximal allelic common subsequence (MACS)
如果對你有幫助，請點個贊！謝謝！