僅用不到150行代碼，我開發出了一個搜索引擎

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"全文搜索無處不在。在Scribd(一個文檔分享平臺)上搜索一本書，在Netflix上搜索一部電影，在亞馬遜上搜索衛生紙商品，或者通過谷歌搜索東西，你都在搜索大量的非結構化數據。更令人感到驚奇地是，即使你搜索的是數百萬(或數十億)條記錄，也能夠獲得毫秒級的響應體驗。在這篇文章中，我們將探索全文搜索引擎的基本組件，並用它們來構建一個可以搜索數百萬個文檔、根據相關性對文檔進行排名的搜索引擎。我們將用不到150行的Python代碼來開發這個搜索引擎。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"數據"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這篇文章中所有的代碼都可以在Github上找到（"},{"type":"link","attrs":{"href":"https:\/\/github.com\/bartdegoede\/python-searchengine\/?fileGuid=4icAb1em6vATdAA7","title":"","type":null},"content":[{"type":"text","text":"https:\/\/github.com\/bartdegoede\/python-searchengine\/"}]},{"type":"text","text":"）。我將在文章中提供代碼片段和鏈接，你可以嘗試自己運行它們。你可以安裝運行示例所需的組件(pip install -r requirements.txt)，然後運行python run.py（"},{"type":"link","attrs":{"href":"https:\/\/github.com\/bartdegoede\/python-searchengine\/blob\/master\/run.py?fileGuid=4icAb1em6vATdAA7","title":"","type":null},"content":[{"type":"text","text":"https:\/\/github.com\/bartdegoede\/python-searchengine\/blob\/master\/run.py"}]},{"type":"text","text":"）。它會下載所有的數據，並運行帶排名和不帶排名的搜索示例。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在開始構建搜索引擎之前，我們需要一些非結構化的數據。我們將搜索英文維基百科中的文章摘要。維基百科被打包成一個約785MB的壓縮XML文件包，其中包含了約627萬篇摘要。我寫了一個簡單的函數（"},{"type":"link","attrs":{"href":"https:\/\/github.com\/bartdegoede\/python-searchengine\/blob\/master\/download.py?fileGuid=4icAb1em6vATdAA7","title":"","type":null},"content":[{"type":"text","text":"https:\/\/github.com\/bartdegoede\/python-searchengine\/blob\/master\/download.py"}]},{"type":"text","text":"）用來下載XML壓縮包，當然你也可以手動下載這個文件。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"數據準備"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個文件是一個包含所有摘要的大型XML文件。每一個摘要內容都包含在標籤中，看起來大致如下所示(我省略了我們不感興趣的標籤)："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"plain"},"content":[{"type":"text","text":"\n