倒排索引是搜索引擎中常用的算法，主要用來實現full text searching，建立關鍵詞和所在文檔的映射關係，很多強大的功能都建立在此基礎之上，關於Inverted Index的詳盡描述可以看Wikipedia。下面按照自己的想法實現之，只是爲了體會這個數據結構的運作。
todo：如果要搜完整的出現一句話如“what is it”可以分別搜這幾個單詞然後看出現在同一個文件連續位置的結果即可，集合運算。
package mythought.invertedindex;
import java.io.BufferedReader;
import java.io.FileReader;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
public class InvertedIndex {
// key word <----> doc file
private Map<String, Set<String>> indexs = new HashMap<String, Set<String>>();
// 這裏假設都是小文件
public void addFile(String fileName, String content) {
String[] words = content.split(" ");
for (int i = 0; i < words.length; i++) {
String word = words[i];
// only record first appeared position
Set<String> wordIndex = indexs.get(word);
if (wordIndex == null) {
wordIndex = new HashSet<String>();
indexs.put(word, wordIndex);
}
wordIndex.add("("+fileName + "," + i+")");
}
}
public void addFile(String fileName) throws Exception{
BufferedReader br = new BufferedReader(new FileReader(fileName));
try {
StringBuilder sb = new StringBuilder();
String line = br.readLine();
while (line != null) {
sb.append(line);
sb.append(" ");// 每行直接串接
line = br.readLine();
}
this.addFile(fileName, sb.toString());
} catch(Exception e){
e.printStackTrace();
}finally {
br.close();
}
}
public Set<String> search(String keyword) {
Set<String> results = indexs.get(keyword);
return new HashSet<String>(results);
}
public static void main(String[] args) throws Exception{
InvertedIndex test = new InvertedIndex();
test.addFile("file1", "hello fuck world todotodo");
test.addFile("file2", "go to get it if you want it");
test.addFile("C:/data/hello.txt");
System.out.println(test.search("it"));
System.out.println(test.search("you"));
System.out.println(test.search("vonzhou"));
}
}
運行結果：
[(file2,7), (file2,3)]
[(file2,5)]
[(C:/data/hello.txt,4)]
參考：