IK Analyzer實現中文分詞

IK Analyzer是基於lucene實現的分詞開源框架，下載路徑:http://code.google.com/p/ik-analyzer/downloads/list

需要在項目中引入：

IKAnalyzer.cfg.xml

IKAnalyzer2012.jar

lucene-core-3.6.0.jar

stopword.dic

什麼都不用改

示例代碼如下(使用IK Analyzer): 

package com.haha.test;


import java.io.IOException;
import java.io.StringReader;
import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
import org.wltea.analyzer.lucene.IKAnalyzer;


public class Test2 {
public static void main(String[] args) throws IOException {
String text="基於java語言開發的輕量級的中文分詞工具包";
//創建分詞對象
Analyzer anal=new IKAnalyzer(true);
StringReader reader=new StringReader(text);
//分詞
TokenStream ts=anal.tokenStream("", reader);
CharTermAttribute term=ts.getAttribute(CharTermAttribute.class);
//遍歷分詞數據
while(ts.incrementToken()){
System.out.print(term.toString()+"|");
}
reader.close();
System.out.println();
}


}

另一種：

package com.haha.test;


import java.io.IOException;
import java.io.StringReader;


import org.wltea.analyzer.core.IKSegmenter;
import org.wltea.analyzer.core.Lexeme;


public class Test3 {

public static void main(String[] args) throws IOException {
String text="基於java語言開發的輕量級的中文分詞工具包";
StringReader sr=new StringReader(text);
IKSegmenter ik=new IKSegmenter(sr, true);
Lexeme lex=null;
while((lex=ik.next())!=null){
System.out.print(lex.getLexemeText()+"|");
}
}


}