flink打包程序提交任務示例

工具

maven、idea、flink1.9-2.11scala

代碼

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.hctang.flink</groupId>
    <artifactId>firstcode</artifactId>
    <version>1.0-SNAPSHOT</version>
    <dependencies>


    <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-scala -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-java</artifactId>
            <version>1.9.0</version>

            <scope>provided</scope>
            <!--<scope>provided</scope>-->

        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_2.11</artifactId>
            <version>1.9.0</version>
            <scope>provided</scope>
            <!--指定包的作用域，集羣中運行的話，很多東西並不需要，-->
         </dependency>

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-scala_2.11</artifactId>
        <version>1.9.0</version>
        <scope>provided</scope>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-streaming-scala -->
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-streaming-scala_2.11</artifactId>
        <version>1.9.0</version>
        <scope>provided</scope>
    </dependency>


    </dependencies>
<build>
        <plugins>
            <!-- 編譯插件 -->
            <plugin>scala
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.6.0</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <encoding>UTF-8</encoding>
                </configuration>
            </plugin>
            <!-- scala編譯插件 -->
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.1.6</version>
                <configuration>
                    <scalaCompatVersion>2.11</scalaCompatVersion>
                    <scalaVersion>2.11.8</scalaVersion>
                    <encoding>UTF-8</encoding>
                </configuration>
                <executions>
                    <execution>
                        <id>compile-scala</id>
                        <phase>compile</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>test-compile-scala</id>
                        <phase>test-compile</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <!-- 打jar包插件(會包含所有依賴) -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.6</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                    <archive>
                        <manifest>
                            <!-- 可以設置jar包的入口類(可選)也可在運行時候動態指定 -->
                            <mainClass>hctang.tech.socketWindowwordCountScala</mainClass>
                        </manifest>
                    </archive>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>



</project>

socketWindowwordCountScala

package hctang.tech

import org.apache.flink.api.java.utils.ParameterTool
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.streaming.api.windowing.time.Time

object socketWindowwordCountScala {
  def main(args: Array[String]): Unit = {
    //獲取socket端口號
    val port:Int=try{
      ParameterTool.fromArgs(args).getInt("port")
    }catch {
      case e:Exception=>{
        System.err.println("No port set. use default port 9000--scala")
      }
        9000
    }

    val env:StreamExecutionEnvironment=StreamExecutionEnvironment.getExecutionEnvironment
    val text=env.socketTextStream("localhost",port,'\n')
    import org.apache.flink.api.scala._
    val windowCounts=text.flatMap(line =>line.split("\\s"))
      .map(w => WordWithCount(w,1))
      .keyBy("word")//分組
      .timeWindow(Time.seconds(2),Time.seconds(1))
      .sum("count");

    windowCounts.print.setParallelism(1);
    windowCounts
    env.execute("Socket window count")




  }
  case class WordWithCount(word: String,count: Long)

}

啓動或關閉集羣

./FLINK_HOME/bin/start-cluster.sh
./FLINK_HOME/bin/stop-cluster.sh

打包

Alt+F12打開idea的終端輸入maven clean package -DskipTests
在工程的target目錄下能看到打包好的文件
如：firstcode-1.0-SNAPSHOT-jar-with-dependencies.jar

提交

./FLINK_HOME/bin/flink run /home/tanghc/IdeaProjects/firstcode/target/firstcode-1.0-SNAPSHOT-jar-with-dependencies.jar
#動態指定

./FLINK_HOME/bin/flink run -c  hctang.tech.socketWindowwordCountScala /home/tanghc/IdeaProjects/firstcode/target/firstcode-1.0-SNAPSHOT-jar-with-dependencies.jar

查看輸出

tail -f FLINK_HOME/log/flink-tanghc-taskexecutor-0-tanghc-X550JX.out