Apache Flink本地模式部署

Apache Flink部署模式有好幾種,本文主要介紹Apache Flink的本地部署模式。
本地部署模式主要用於開發者程序調試測試使用。

先決條件

  • 運行系統:系統方面沒有過多要求,Linux、Mac、Windows均可
  • Java 1.8.x以上,Apache Flink不支持1.8.x以下的版本。

檢查系統的Java版本

$ java -version

java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)

下載、啓動Flink

下載

進入Apache Flink官網下載安裝包。下載地址:https://flink.apache.org/downloads.html
我們下載Apache Flink的最新包1.6.2版本

wget http://mirror.metrocast.net/apache/flink/flink-1.6.2/flink-1.6.2-bin-scala_2.11.tgz

解壓

$ tar -zxvf flink-1.6.2-bin-scala_2.11.tgz
x flink-1.6.2/
x flink-1.6.2/bin/
x flink-1.6.2/bin/config.sh
x flink-1.6.2/bin/flink
x flink-1.6.2/bin/flink-console.sh
x flink-1.6.2/bin/flink-daemon.sh
x flink-1.6.2/bin/flink.bat
x flink-1.6.2/bin/historyserver.sh
x flink-1.6.2/bin/jobmanager.sh
x flink-1.6.2/bin/mesos-appmaster-job.sh
x flink-1.6.2/bin/mesos-appmaster.sh
x flink-1.6.2/bin/mesos-taskmanager.sh
x flink-1.6.2/bin/pyflink-stream.sh
x flink-1.6.2/bin/pyflink.bat
x flink-1.6.2/bin/pyflink.sh
x flink-1.6.2/bin/sql-client.sh
x flink-1.6.2/bin/standalone-job.sh
x flink-1.6.2/bin/start-cluster.bat
x flink-1.6.2/bin/start-cluster.sh
x flink-1.6.2/bin/start-scala-shell.sh
x flink-1.6.2/bin/start-zookeeper-quorum.sh
x flink-1.6.2/bin/stop-cluster.sh
x flink-1.6.2/bin/stop-zookeeper-quorum.sh
x flink-1.6.2/bin/taskmanager.sh
x flink-1.6.2/bin/yarn-session.sh
x flink-1.6.2/bin/zookeeper.sh
x flink-1.6.2/conf/
x flink-1.6.2/conf/flink-conf.yaml
x flink-1.6.2/conf/log4j-cli.properties
x flink-1.6.2/conf/log4j-console.properties
x flink-1.6.2/conf/log4j-yarn-session.properties
x flink-1.6.2/conf/log4j.properties
x flink-1.6.2/conf/logback-console.xml
x flink-1.6.2/conf/logback-yarn.xml
x flink-1.6.2/conf/logback.xml
x flink-1.6.2/conf/masters
x flink-1.6.2/conf/slaves
x flink-1.6.2/conf/sql-client-defaults.yaml
x flink-1.6.2/conf/zoo.cfg
x flink-1.6.2/examples/
x flink-1.6.2/examples/batch/
x flink-1.6.2/examples/batch/ConnectedComponents.jar
x flink-1.6.2/examples/batch/DistCp.jar
x flink-1.6.2/examples/batch/EnumTriangles.jar
x flink-1.6.2/examples/batch/KMeans.jar
x flink-1.6.2/examples/batch/PageRank.jar
x flink-1.6.2/examples/batch/TransitiveClosure.jar
x flink-1.6.2/examples/batch/WebLogAnalysis.jar
x flink-1.6.2/examples/batch/WordCount.jar
x flink-1.6.2/examples/gelly/
x flink-1.6.2/examples/gelly/flink-gelly-examples_2.11-1.6.2.jar
x flink-1.6.2/examples/python/
x flink-1.6.2/examples/python/batch/
x flink-1.6.2/examples/python/batch/TPCHQuery10.py
x flink-1.6.2/examples/python/batch/TPCHQuery3.py
x flink-1.6.2/examples/python/batch/TriangleEnumeration.py
x flink-1.6.2/examples/python/batch/WebLogAnalysis.py
x flink-1.6.2/examples/python/batch/WordCount.py
x flink-1.6.2/examples/python/batch/__init__.py
x flink-1.6.2/examples/python/streaming/
x flink-1.6.2/examples/python/streaming/fibonacci.py
x flink-1.6.2/examples/python/streaming/word_count.py
x flink-1.6.2/examples/streaming/
x flink-1.6.2/examples/streaming/IncrementalLearning.jar
x flink-1.6.2/examples/streaming/Iteration.jar
x flink-1.6.2/examples/streaming/Kafka010Example.jar
x flink-1.6.2/examples/streaming/SessionWindowing.jar
x flink-1.6.2/examples/streaming/SocketWindowWordCount.jar
x flink-1.6.2/examples/streaming/StateMachineExample.jar
x flink-1.6.2/examples/streaming/TopSpeedWindowing.jar
x flink-1.6.2/examples/streaming/Twitter.jar
x flink-1.6.2/examples/streaming/WindowJoin.jar
x flink-1.6.2/examples/streaming/WordCount.jar
x flink-1.6.2/lib/
x flink-1.6.2/lib/flink-dist_2.11-1.6.2.jar
x flink-1.6.2/lib/flink-python_2.11-1.6.2.jar
x flink-1.6.2/lib/log4j-1.2.17.jar
x flink-1.6.2/lib/slf4j-log4j12-1.7.7.jar
x flink-1.6.2/LICENSE
x flink-1.6.2/log/
x flink-1.6.2/NOTICE
x flink-1.6.2/opt/
x flink-1.6.2/opt/flink-cep-scala_2.11-1.6.2.jar
x flink-1.6.2/opt/flink-cep_2.11-1.6.2.jar
x flink-1.6.2/opt/flink-gelly-scala_2.11-1.6.2.jar
x flink-1.6.2/opt/flink-gelly_2.11-1.6.2.jar
x flink-1.6.2/opt/flink-metrics-datadog-1.6.2.jar
x flink-1.6.2/opt/flink-metrics-dropwizard-1.6.2.jar
x flink-1.6.2/opt/flink-metrics-ganglia-1.6.2.jar
x flink-1.6.2/opt/flink-metrics-graphite-1.6.2.jar
x flink-1.6.2/opt/flink-metrics-prometheus-1.6.2.jar
x flink-1.6.2/opt/flink-metrics-slf4j-1.6.2.jar
x flink-1.6.2/opt/flink-metrics-statsd-1.6.2.jar
x flink-1.6.2/opt/flink-ml_2.11-1.6.2.jar
x flink-1.6.2/opt/flink-queryable-state-runtime_2.11-1.6.2.jar
x flink-1.6.2/opt/flink-s3-fs-hadoop-1.6.2.jar
x flink-1.6.2/opt/flink-s3-fs-presto-1.6.2.jar
x flink-1.6.2/opt/flink-sql-client-1.6.2.jar
x flink-1.6.2/opt/flink-streaming-python_2.11-1.6.2.jar
x flink-1.6.2/opt/flink-swift-fs-hadoop-1.6.2.jar
x flink-1.6.2/opt/flink-table_2.11-1.6.2.jar
x flink-1.6.2/README.txt

配置環境變量

在~/.bash_profile文件中添加以下變量:

export $FLINK_HOME=/opt/flink-1.6.2
export $PATH = $PATH:$FLINK_HOME/bin

啓動Flink

$ $FLINK_HOME/bin/start-cluster.sh  # Start Flink

通過web前端檢查服務是否正常運行
http://localhost:8081
Web前端可以看到有一個可用的TaskManager實例。

在這裏插入圖片描述

通過jps檢查服務啓動情況

$ jps
4705 TaskManagerRunner
4940 Jps
4286 StandaloneSessionClusterEntrypoint

還可以通過檢查logs目錄中的日誌文件來驗證系統是否正在運行:

flink-1.6.2  tail log/flink-*-standalonesession-*.log
2018-11-16 09:22:03,204 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Rest endpoint listening at localhost:8081
2018-11-16 09:22:03,205 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - http://localhost:8081 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000
2018-11-16 09:22:03,205 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Web frontend listening at http://localhost:8081.
2018-11-16 09:22:03,359 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/resourcemanager .
2018-11-16 09:22:03,396 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher .
2018-11-16 09:22:03,441 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - ResourceManager akka.tcp://flink@localhost:6123/user/resourcemanager was granted leadership with fencing token 00000000000000000000000000000000
2018-11-16 09:22:03,442 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Starting the SlotManager.
2018-11-16 09:22:03,476 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Dispatcher akka.tcp://flink@localhost:6123/user/dispatcher was granted leadership with fencing token 00000000-0000-0000-0000-000000000000
2018-11-16 09:22:03,483 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Recovering all persisted jobs.
2018-11-16 09:22:04,506 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - Registering TaskManager with ResourceID c5c7f1655d16bb0f18743804f4b1c72a (akka.tcp://[email protected]:55045/user/taskmanager_0) at ResourceManager

運行example

首先,我們使用netcat來啓動本地服務器

$ nc -l 9000

提交Flink工程

$ ./bin/flink run examples/streaming/SocketWindowWordCount.jar --port 9000
Starting execution of program

程序連接到socket端口並等待輸入。 可以檢查Web界面以驗證作業是否按預期運行:
在這裏插入圖片描述
在這裏插入圖片描述

檢查jps,多了一個CliFrontend進程。

4705 TaskManagerRunner
5330 Jps
5052 CliFrontend
4286 StandaloneSessionClusterEntrypoint

####單詞在5秒的時間窗口中計算(處理時間,翻滾窗口)並打印到標準輸出。 監視TaskManager的輸出文件並在nc中寫入一些文本(輸入在點擊後逐行發送到Flink):

$ nc -l 9000
lorem ipsum
ipsum ipsum ipsum
bye

.out文件會在每個時間窗口的末尾打印計數

$ tail -f log/flink-*-taskexecutor-*.out
lorem : 1
bye : 1
ipsum : 4

停止Flink服務

$ ./bin/stop-cluster.sh

關注公衆號

微信公衆號(SZBigdata-Club):後續博客的文檔都會轉到微信公衆號中。
1、公衆號會持續給大家推送技術文檔、學習視頻、技術書籍、數據集等。
2、接受大家投稿支持。
3、對於各公司hr招聘的,可以私下聯繫我,把招聘信息發給我我會在公衆號中進行推送。
在這裏插入圖片描述

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章