關於Ontonotes5.0數據集下載過程（個人向）

前言

說實話，這個數據集下載真的是很折騰了很久，這篇文章僅僅是介紹獲取OntoNotes Release 5.0數據集的全過程，以及對於指代消解中英文數據的預處理。

獲取數據

1.首先要在官網上註冊賬號。除了郵箱和組織需要注意下，其他無所謂了。

2. 註冊完畢之後登陸，會提示：You currently have a guest account with LDC. If you’d like to be able to access more data please consider a membership for your organization.
   此時，理論上是要等待組織管理員通過你的申請，才能夠有權限下載免費的數據。不然也只能乾瞪眼。
   但是！實際情況是等了很久都沒人理你，然後不了了之。這時候，你需要用你註冊的郵箱發送一份郵件給官方，大意是描述等了一段時間沒人響應你的申請，可能是由於管理員畢業、離校之類的種種原因，希望得到幫助。
   然後過一兩天（時差問題）你就會收到官方的回覆，告訴你組織管理員的郵箱和電話（如果你註冊數據填的不夠完備，她會先告訴你哪些內容需要補充）。
   我們學校的管理員是個大佬，我只敢發郵件去叨擾他，索性的是很快就得到了回覆，通過了我的申請。
   

3. 再重新登錄，你就會發現你已經是個有組織的人~~在downloads中可以看到可以下載的內容。

4. 一般情況下，下載中是沒有你想要的數據集的，你需要搜索到對應的數據集然後下單，過幾天之後才能在你的下載頁面看到對應內容的下載。
   

數據處理

以下方法適用於獲取指代消解的數據集

1. 目前使用這個項目的處理腳本去預處理數據的。值得注意的是代碼是__python2__寫的，而且需要的依賴還挺多，不過改成python3也很容易，報錯的地方修改下即可（因爲一般都是拋出異常和print語法不對而已）
   

2. 執行完畢之後會在同目錄生成三個json文件數據：訓練集-驗證集-測試集。
   其中doc_key爲文本分類，sentences爲切分完畢的對話，speakers爲句子發言者、clusters即指代文本短語的聚類結果(共指)
   英文：train.english.jsonlines、dev.english.jsonlines、test.english.jsonlines
   中文：train.chinese.jsonlines、dev.chinese.jsonlines、test.chinese.jsonlines
   示例數據如下：

{"doc_key": "bc/cctv/00/cctv_0001_0", "sentences": [["What", "kind", "of", "memory", "?"], ["We", "respectfully", "invite", "you", "to", "watch", "a", "special", "edition", "of", "Across", "China", "."], ["WW", "II", "Landmarks", "on", "the", "Great", "Earth", "of", "China", ":", "Eternal", "Memories", "of", "Taihang", "Mountain"], ["Standing", "tall", "on", "Taihang", "Mountain", "is", "the", "Monument", "to", "the", "Hundred", "Regiments", "Offensive", "."], ["It", "is", "composed", "of", "a", "primary", "stele", ",", "secondary", "steles", ",", "a", "huge", "round", "sculpture", "and", "beacon", "tower", ",", "and", "the", "Great", "Wall", ",", "among", "other", "things", "."], ["A", "primary", "stele", ",", "three", "secondary", "steles", ",", "and", "two", "inscribed", "steles", "."], ["The", "Hundred", "Regiments", "Offensive", "was", "the", "campaign", "of", "the", "largest", "scale", "launched", "by", "the", "Eighth", "Route", "Army", "during", "the", "War", "of", "Resistance", "against", "Japan", "."], ["This", "campaign", "broke", "through", "the", "Japanese", "army", "'s", "blockade", "to", "reach", "base", "areas", "behind", "enemy", "lines", ",", "stirring", "up", "anti-Japanese", "spirit", "throughout", "the", "nation", "and", "influencing", "the", "situation", "of", "the", "anti-fascist", "war", "of", "the", "people", "worldwide", "."], ["This", "is", "Zhuanbi", "Village", ",", "Wuxiang", "County", "of", "Shanxi", "Province", ",", "where", "the", "Eighth", "Route", "Army", "was", "headquartered", "back", "then", "."], ["On", "a", "wall", "outside", "the", "headquarters", "we", "found", "a", "map", "."], ["This", "map", "was", "the", "Eighth", "Route", "Army", "'s", "depiction", "of", "the", "Mediterranean", "Sea", "situation", "at", "that", "time", "."], ["This", "map", "reflected", "the", "European", "battlefield", "situation", "."], ["In", "1940", ",", "the", "German", "army", "invaded", "and", "occupied", "Czechoslovakia", ",", "Poland", ",", "the", "Netherlands", ",", "Belgium", ",", "and", "France", "."], ["It", "was", "during", "this", "year", "that", "the", "Japanese", "army", "developed", "a", "strategy", "to", "rapidly", "force", "the", "Chinese", "people", "into", "submission", "by", "the", "end", "of", "1940", "."], ["In", "May", ",", "the", "Japanese", "army", "launched", "--"], ["From", "one", "side", ",", "it", "seized", "an", "important", "city", "in", "China", "called", "Yichang", "."], ["Um", ",", ",", "uh", ",", "through", "Yichang", ",", "it", "could", "directly", "reach", "Chongqing", "."], ["Ah", ",", "that", "threatened", "Chongqing", "."], ["Then", "they", "would", ",", "ah", ",", "bomb", "these", "large", "rear", "areas", "such", "as", "Chongqing", "."], ["So", ",", "along", "with", "the", "coordinated", ",", "er", ",", "economic", "blockade", ",", "military", "offensives", ",", "and", "strategic", "bombings", ",", "er", ",", "a", "simultaneous", "attack", "was", "launched", "in", "Hong", "Kong", "to", "lure", "the", "KMT", "government", "into", "surrender", "."], ["The", "progress", "of", "this", "coordinated", "offensive", "was", "already", "very", "entrenched", "by", "then", "."]], "speakers": [["Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1"], ["Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1"], ["Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1"], ["Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1"], ["Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1"], ["Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1"], ["Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1"], ["Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1"], ["Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1"], ["Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1"], ["Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1"], ["Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1"], ["Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1"], ["Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1", "Speaker#1"], ["Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang"], ["Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang"], ["Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang"], ["Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang"], ["Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang"], ["Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang"], ["Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang", "Luo_huanzhang"]], "clusters": [[[135, 136], [273, 273], [26, 26]], [[36, 37], [31, 32]], [[113, 114], [42, 45], [88, 91]], [[39, 45], [47, 47]], [[75, 77], [51, 53]], [[55, 56], [79, 81]], [[185, 189], [162, 165], [101, 104]], [[285, 285], [267, 267], [298, 298], [258, 260], [117, 120], [235, 237]], [[200, 201], [179, 180], [182, 183]], [[232, 233], [209, 209], [253, 253]], [[283, 283], [269, 275]], [[288, 288], [293, 293]], [[295, 295], [289, 289], [310, 310]]]}

{"doc_key": "bn/ctv/01/ctv_0156_0", "sentences": [["投資", "１０億", "元", "興建", "的", "廣東", "長龍", "夜間", "動物", "世界", "２８號", "在", "廣州", "建成", "開放", "。"], ["這", "是", "繼", "新加坡", "之後", "，", "世界", "上", "的", "第二", "個", "夜間", "動物園", "。"], ["中國", "第一", "家", "夜間", "動物園", "坐落", "在", "廣州", "番禺", "，", "佔地", "３０００", "畝", "，", "是", "世界", "上", "規模", "最", "大", "的", "夜間", "動物園", "，", "夜間", "活動", "是", "動物", "的", "基本", "特徵", "，", "夜間", "看", "動物", "比", "白天", "更爲", "精彩", "。"], ["根據", "動物", "的", "活動", "習性", "動物園", "每", "天", "晚上", "６：３０", "開始", "對", "遊人", "開放", "至", "到", "零點", "。"], ["乘客", "乘座", "小", "火車", "可以", "觀察", "到", "中華", "山地", "、", "南美", "河谷", "、", "南非", "高原", "、", "印度", "森林", "等", "園中", "３００多", "種", "近", "萬", "只", "習慣", "夜出", "的", "動物", "。"], ["叢林", "、", "原野", "和", "動物", "吼叫聲", "讓", "人", "彷彿", "置身", "於", "原始", "森林", "之中", "。"], ["廣東臺", "記者", "報道", "。"]], "speakers": [["-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-"], ["-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-"], ["-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-"], ["-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-"], ["-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-"], ["-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-"], ["-", "-", "-", "-"]], "clusters": [[[0, 9], [30, 34], [16, 16], [75, 75]]]}

3. 如果需要獲取文本對應的詞彙表（字爲單位），保留第一步腳本中的python get_char_vocab.py，他會在當前目錄生成char_vocab.english.txt/char_vocab.chinese.txt。

後記

以上內容僅僅是爲了記錄個人獲取到這份數據的整個過程。由於版權問題和組織協議，不可能會把數據集發出來的，所以希望評論不要出現任何留言想直接要數據集的，打錢都不行~~