Hadoop windows編譯:
可以略過這一段,直接下載hadoo2.7.1 bin文件,我的資源裏有最終生成的winutils.exe和一些native code,放在bin目錄就行了
參考building.txt以及http://wiki.apache.org/hadoop/Hadoop2OnWindows,編譯條件:
* Windows System - win8
* JDK 1.7+ - jdk1.8
* Maven 3.0 or later - maven 3.3
* Findbugs 1.3.9 (if running findbugs) - None
* ProtocolBuffer 2.5.0 - 2.5 build by vs2013
* CMake 2.6 or newer - cmake3.3
* Windows SDK 7.1 or Visual Studio 2010 Professional - vs2010
* Windows SDK 8.1 (if building CPU rate control for the container executor)
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These
tools must be present on your PATH.
1. Unix command-line tools: http://git-scm.com/download/win
PATH環境變量設置C:\Program Files (x86)\Git\cmd;C:\Program Files (x86)\Git\bin
想用gnuwin32的話,hosts文件需要添加(國內訪問不了)
216.34.181.96 getgnuwin32.sourceforge.net
2. protocolbuffer 2.5.0 可以用vs2010/vs2013編譯,幾個工程需要加上 /FS
加上,#include <algorithm>
編譯好的protoc.exe / libprotobuf.lib / libprotobuf-lite.lib / libprotoc.lib 放到c:/windows,打開CMD輸入protoc,
3. 拷貝zlib1.dll(網上可以下載到,最好用1.2.7版本的)到C:/windows
4. 按編譯過程中提示的錯誤信息,把C:\Java\jdk1.8.0_51\lib\tools.jar拷貝到C:\Java\jdk1.8.0_51\lib\
5. 打開CMD,注意JAVA_HOME目錄中不要有Program file這樣有空格
CALL "%VS100COMNTOOLS%..\..\VC\bin\vcvars32.bat"
set Platform=Win32
set JAVA_HOME=C:\Java\jdk1.8.0_51
cd e:/hadoop-2.7.1-src
mvn package -Pdist,native-win -DskipTests -Dtar
6. mvn可以多試幾次,最終tar文件在hadoop-dist\target\
啓動Hadoop:
配置hadoop可以完全參照linux下配置,需要注意的是:
1. 涉及到本機目錄的地方,加上file:///,否則 hdfs namenode -format 會失敗
2. 可能會提示Error: missing `server' JVM at `C:\Java\jdk1.8.0_51\bin\server\jvm.dll'.
將C:\Java\jdk1.8.0_51\jre\bin\server 目錄拷貝到C:\Java\jdk1.8.0_51\bin\
eclipse導入Nutch2.3:
1. windows下編譯,ant runtime和linux差別不大
2. 導入nutch2.3,網上有例程。ant runtime以後再導入例程
打開eclipse,new - java project - ,基本就
可以finish了,還會有一些bug:
3. Run as - java appliction, 選擇InjectorJob
4. 運行成功,先運行hadoop
5. InjectorJob: java.lang.NullPointerException
at java.lang.ProcessBuilder.start(Unknown Source)
,同時HADOOP_HOME/bin加入到PATH可以解決
或許需要重啓一下eclipse:)