Flink 在 idea上提交任務到遠程服務器

Flink自身提供了遠程提交任務的環境,源碼如下:

請查看StreamExecutionEnvironment 類中 createRemoteEnvironment 方法

def createRemoteEnvironment(
    host: String,
    port: Int,
    parallelism: Int,
    jarFiles: String*): StreamExecutionEnvironment = {

  val javaEnv = JavaEnv.createRemoteEnvironment(host, port, jarFiles: _*)
  javaEnv.setParallelism(parallelism)
  new StreamExecutionEnvironment(javaEnv)
}

遠程提交示例代碼如下:

package com.flink.remotesubmit

import org.apache.flink.streaming.api.scala._

object RemoteSubmitApp extends App {

  val host: String = "node02"
  val port: Int = 8081
  val jarFiles = "E:\\CDHProjectDemo\\flink-demo\\target\\flink-demo-0.0.1-SNAPSHOT.jar"

  val env = StreamExecutionEnvironment.createRemoteEnvironment(host, port, jarFiles)

  val socketHost: String = "node01"
  val socketPort: Int = 7777
  val socketDs: DataStream[String] = env.socketTextStream(socketHost, socketPort)

  socketDs.flatMap(_.split(" "))
    .map((_, 1))
    .keyBy(0)
    .sum(1)
    .print()

  env.execute("Remote Submit Job")
}

注意:

1. 需要保持代碼和jar一致性，意思就是修改代碼之後需重新執行 mvn clean package
2. 需在項目的 src/main/resource 目錄中添加相關配置文件(core-site.xml、hdfs-site.xml、yarn-site.xml、mapred-site.xml等)

運行情況