Spark 動態上架下架worker

關閉 $SPARK_HOME/sbin/spark-daemon.sh --config conf/ stop org.apache.spark.deploy.worker.Worker 1 --webui-port 8081 spark://llc1:7077

開啓 $SPARK_HOME/sbin/spark-daemon.sh --config conf/ start org.apache.spark.deploy.worker.Worker 1 --webui-port 8081 -c 24 -m 60G spark://llc1:7077

使用命令

$SPARK_HOME/sbin/spark-daemon.sh [--config <conf-dir>] (start|stop|status) <spark-command> <spark-instance-number> <args...>

第一個參數　：　–config $SPARK_HOME/conf
第二個參數　：　start|stop|status
第三個參數　：　org.apache.spark.deploy.worker.Worker（worker類的路徑）
第四個參數　：　這個worker的號碼，根據機器上已有的worker數來看
第五個參數　：　啓動時的參數，下面是源碼解析參數類　WorkerArguments.scala　中截取，都很清楚，傳自己需要的參數即可

private def parse(args: List[String]): Unit = args match {
  case ("--ip" | "-i") :: value :: tail =>
    Utils.checkHost(value, "ip no longer supported, please use hostname " + value)
    host = value
    parse(tail)

  case ("--host" | "-h") :: value :: tail =>
    Utils.checkHost(value, "Please use hostname " + value)
    host = value
    parse(tail)

  case ("--port" | "-p") :: IntParam(value) :: tail =>
    port = value
    parse(tail)

  case ("--cores" | "-c") :: IntParam(value) :: tail =>
    cores = value
    parse(tail)

  case ("--memory" | "-m") :: MemoryParam(value) :: tail =>
    memory = value
    parse(tail)

  case ("--work-dir" | "-d") :: value :: tail =>
    workDir = value
    parse(tail)

  case "--webui-port" :: IntParam(value) :: tail =>
    webUiPort = value
    parse(tail)

  case ("--properties-file") :: value :: tail =>
    propertiesFile = value
    parse(tail)

  case ("--help") :: tail =>
    printUsageAndExit(0)

  case value :: tail =>
    if (masters != null) {  // Two positional arguments were given
      printUsageAndExit(1)
    }
    masters = Utils.parseStandaloneMasterUrls(value)
    parse(tail)

  case Nil =>
    if (masters == null) {  // No positional argument was given
      printUsageAndExit(1)
    }

  case _ =>
    printUsageAndExit(1)
}

一個例子

$SPARK_HOME/sbin/spark-daemon.sh --config conf/ start org.apache.spark.deploy.worker.Worker 1 --webui-port 8082 -c 4 -m 9G spark://llc1:7077