Flink集羣搭建與一鍵腳本啓停

Flink集羣搭建

由於最近項目上需要使用到flink，所以特地學習了一下，首先還是介紹搭建步驟

環境準備

本次安裝集羣需要3個節點，下面是我的節點信息：

1. node1 192.168.75.200 master
2. node2 192.168.75.201 slave
3. node3 193.168.75.202 slave

安裝包下載

鏈接：https://pan.baidu.com/s/1qde-fH8ndJhZVAQHWsMG2A
提取碼：4tul

安裝Hadoop

參考這條帖子：https://blog.csdn.net/IUNIQUE/article/details/84297695

相關配置文件修改

在master節點上面進行相關配置文件的修改

配置slaves

vi slaves

文本中寫入下面兩條數據,指定task節點：

node2
node3

配置masters

vi masters

文本中寫入下面內容，指定job節點

node1:8078

配置flink-conf.yaml

#==================================================================
# Common
#==================================================================
#指定jobmanager的ip地址
jobmanager.rpc.address: node1  
# The RPC port where the JobManager is reachable.
jobmanager.rpc.port: 6123
# The heap size for the JobManager JVM
jobmanager.heap.size: 1024m
# The heap size for the TaskManager JVM
taskmanager.heap.size: 1024m
# The number of task slots that each TaskManager offers. Each slot runs one parallel pipeline.
#taskmanager上面的slots個數
taskmanager.numberOfTaskSlots: 3
# The parallelism used for programs that did not specify and other parallelism.
#默認的並行度
parallelism.default: 3

#=================================================================
# Web Frontend
#=================================================================
# The address under which the web-based runtime monitor listens.
#web.address: 0.0.0.0
# The port under which the web-based runtime monitor listens.
# A value of -1 deactivates the web server.
#web ui 監控界面
rest.port: 8078 
# Flag to specify whether job submissi

#==================================================================
# HistoryServer
#==================================================================
# The HistoryServer is started and stopped via bin/historyserver.sh (start|stop)
# Directory to upload completed jobs to. Add this directory to the list of
# monitored directories of the HistoryServer as well (see below).
#設置jobmanager文件存放路徑
jobmanager.archive.fs.dir: hdfs://node1:9000/var/log/hadoop-flink
# The address under which the web-based HistoryServer listens.
#historyserver.web.address: 0.0.0.0
# The port under which the web-based HistoryServer listens.
#設置historyserver web port
historyserver.web.port: 18078
# Comma separated list of directories to monitor for completed jobs.
#設置historyserver文件存放路徑
historyserver.archive.fs.dir: hdfs://node1:9000/var/log/hadoop-flink

安裝包分發

cd /root/packages/
scp flink1.7.2 root@node2:`pwd`
scp flink1.7.2 root@node3:`pwd`

一鍵啓停腳本

start-flink.sh

#!/bin/bash
cd /root/packages/flink1.7.2/bin
./start-cluster.sh 

stop-flink.sh

#!/bin/bash
cd /root/packages/flink1.7.2/bin
./stop-cluster.sh

web監控界面

http://192.168.75.200:8078