首先感謝wilco的文章: http://www.cnblogs.com/e-life/p/3897881.html
一、環境準備
首先，配置開發環境
需要的環境有JDK1.7，myeclipse,svn,ant一級myeclipse下的兩個插件subeclipse和ivyDe,
下載地址 http://subeclipse.trigris.org/update_1.8.x
http://www.apache.org/dist/ant/ivyde/updatesite
二、從SVN檢出項目,地址
https://svn.apache.org/repos/asf/nutch/tags/release-2.2.1
Finish完成導入.
三、修改ivy目錄下的ivysettings.xml地址http://mirrors.ibiblio.org/maven2/ (試了下,這個還能訪問到)
四、修改ivy目錄下的ivy.xml(增加mysql訪問以來java包)
修改gora-core版本爲0.2.1,並解除註釋gora-sql和mysql-connector-java
五、到項目根目錄執行 ant eclipse
一開始會有如下問題:
具體解決辦法是到ivy官網：http://ant.apache.org/ivy/index.html下載ivy-*-rc2.jar,複製到ant的lib目錄下,再次執行 ant eclipse,就不會報錯了,執行時間比較長,這是由於ivy要下載jar包,執行結果如下
六、回到myeclipse,刷新項目,會發現目錄結果已經發生變化
七、工程右鍵Build Path-->Config Build Path-->Order and Export下選中Conf文件置頂
八、修改Conf文件夾下gora.properties配置mysql
九、工程目錄下新建文件夾urls,url目錄下新建一文件url,裏面輸入要爬取的root_url,比如:http://www.163.com
十、配置conf目錄下nutch-site.xml
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>http.agent.name</name> <value>YourNutchSpider</value> </property> <property> <name>http.accept.language</name> <value>ja-jp, en-us,en-gb,en,zh-cn,zh-tw;q=0.7,*;q=0.3</value> <description>Value of the “Accept-Language” request header field. This allows selecting non-English language as default one to retrieve. It is a useful setting for search engines build for certain national group. </description> </property> <property> <name>parser.character.encoding.default</name> <value>utf-8</value> <description>The character encoding to fall back to when no other information is available </description> </property> <property> <name>plugin.folders</name> <value>src/plugin</value> <description>Directories where nutch plugins are located. Each element may be a relative or absolute path. If absolute, it is used as is. If relative, it is searched for on the classpath. </description> </property> </pre> <pre name="code" class="html"><!-- 爲解決utf-8類中的空指針問題 --> <property> <span> </span> <name>generate.batch.id</name> <span> </span> <value>*</value> </property> <property> <name>storage.data.store.class</name> <value>org.apache.gora.sql.store.SqlStore</value> <description>The Gora DataStore class for storing and retrieving data. Currently the following stores are available: …. </description> </property> </configuration>
十一、配置完上述步驟後配置命令運行Run Configuation選擇org.apache.nutch.crawl.Crawler,參數設置urls-depth 3 -topN 5 和Dhadoop.log.dir=logs-Dhadoop.file=hadoop.log
這時候運行會遇到錯誤Exception in thread "main" java.io.IOException:Failed to set permissions of Path:\tmp\hadoop-Administrator\mapred\staging\Administrator238829409\.staging to 0700
十二、上述錯誤一般只是在windows下才會遇到的,所以我們一般會去修改Haddop-core-1.2.0源碼中的org.apache.hadoop.hadoop.fs下的FileUtil.java修改其中的CheckReturnValue方法,註釋掉其中的內容
private static void checkReturnValue(boolean rv, File p,
FsPermission permission ) throws IOException {// if (!rv) {// throw new IOException("Failed to set permissions of path: " + p + // " to " + // String.format("%04o", permission.toShort()));// }
}然後編譯成java包替換我們工程build/lib下的hadoop-core-1.2.0.jar.
另外一種方法是找到FileUtil.java編譯過後的class文件,替換掉jar包中相應的class文件FileUtil.class和FileUtil$CygPathCommand.class(用壓縮軟件打開直接替換即可)
十三、接下來在運行就沒有問題
你會發現mysql數據庫中很多記錄除了id，status這幾個基本字段有數據外其他值都是null，這是正常情況，請注意status字段，
status 記錄抓取狀態
1 unfetched (links not yet fetched due to limits set in regex-urlfilter.txt, -TopN crawl parameters, etc.)
2 fetched (page was successfully fetched)
3 gone (that page no longer exists)
4 redir_temp (temporary redirection — see reprUrl below for more details)
5 redir_perm (permanent redirection — see reprUrl below for more details)
34 retry
38 not modified
這是因爲我們的topN設置太小，所以很多url被限制抓取了（當然還有可能是url被regex-urllfilter過濾了），自己注意一下就好了。到此我們就完成了整個流程的配置和運行。