windows下使用maven編譯spark源碼

1.環境

jdk1.8

scala2.11.8

spark2.11

maven4.0

2.github客戶端

官方給出的命令是在Linux上的

./build/mvn -DskipTests clean package

調用shell腳本取只想一些命令然後在用maven編譯,一些過程如下

install_app() {
  local remote_tarball="$1/$2"
  local local_tarball="${_DIR}/$2"
  local binary="${_DIR}/$3"

  # setup `curl` and `wget` silent options if we're running on Jenkins
  local curl_opts="-L"
  local wget_opts=""
  if [ -n "$AMPLAB_JENKINS" ]; then
    curl_opts="-s ${curl_opts}"
    wget_opts="--quiet ${wget_opts}"
  else
    curl_opts="--progress-bar ${curl_opts}"
    wget_opts="--progress=bar:force ${wget_opts}"
  fi

  if [ -z "$3" -o ! -f "$binary" ]; then
    # check if we already have the tarball
    # check if we have curl installed
    # download application
    [ ! -f "${local_tarball}" ] && [ $(command -v curl) ] && \
      echo "exec: curl ${curl_opts} ${remote_tarball}" 1>&2 && \
      curl ${curl_opts} "${remote_tarball}" > "${local_tarball}"
    # if the file still doesn't exist, lets try `wget` and cross our fingers
    [ ! -f "${local_tarball}" ] && [ $(command -v wget) ] && \
      echo "exec: wget ${wget_opts} ${remote_tarball}" 1>&2 && \
      wget ${wget_opts} -O "${local_tarball}" "${remote_tarball}"
    # if both were unsuccessful, exit
    [ ! -f "${local_tarball}" ] && \
      echo -n "ERROR: Cannot download $2 with cURL or wget; " && \
      echo "please install manually and try again." && \
      exit 2
........

需要用github的gitbush調用shell腳本

github客戶端安裝

鼠標右鍵會出現打開gitbush。。。

3.編譯

在項目根目錄下右鍵打開gitbush 輸入命令

./build/mvn -DskipTests clean package

如果在有maven的報錯可能是maven配置問題，但是shell腳本執行的操作已經完成

直接在根目錄下maven打包就可以了

mvn install

編譯時間很長