目錄
Redis服務
在服務器上搭建redis服務，本次採用redis-5.0.8，官網下載gz包，tar解壓，再進行make操作，不報錯的話，進入src（出現redis-server、redis-cli文件即爲make成功）目錄進行make install操作。之後進入src之前的目錄修改redis.conf文件，配置爲遠程可以訪問和後臺運行，這樣可以在本地使用可視化工具（比如：RedisDesktopManager）進行查看。
修改配置文件：
配置後臺運行
#daemonize no
daemonize yes
配置遠程訪問
vim redis.conf
註釋
bind 127.0.0.1
protected-mode yes
修改爲
bind 0.0.0.0
protected-mode no
下面是測試結果：
編寫RedusUtil並測試
public static final Jedis jedis = new Jedis("自己redis服務的ip",6379);
public static String getBykey (String key){
return jedis.get(key);
}
public static void main(String[] args) {
jedis.set("test3","test33");
String value = jedis.get("test3");
System.out.println(value);
}
flink整合redis進行測試：
RedisUtil.jedis.lpush("pingdaord:"+pindaoid,count+"");
將本地應用放在服務器上測試
首先下載flink-1.7.2-bin-hadoop28-scala_2.11.tgz，在服務器上解壓並啓動./start-cluster.sh,作爲運行flink進行MapReduce的環境
啓動後可以輸入：http:服務器地址:8081/進入面板頁面查看
由於服務器一直刷不出Task Slots的數量，所以只能進行到這裏，後續是本地調用上報服務，生成數據，在服務器flink上進行分析並存入redis
報錯記錄
org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate all requires slots within timeout of 300000 ms. Slots required: 2, slots allocated: 0
at org.apache.flink.runtime.executiongraph.ExecutionGraph.lambda$scheduleEager$3(ExecutionGraph.java:991)
at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at org.apache.flink.runtime.concurrent.FutureUtils$ResultConjunctFuture.handleCompletedFuture(FutureUtils.java:535)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:772)
at akka.dispatch.OnComplete.internal(Future.scala:258)
at akka.dispatch.OnComplete.internal(Future.scala:256)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603)
at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
at java.lang.Thread.run(Thread.java:748)
2020-05-03 22:51:40,205 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Map (1/1) (f29f5bc1259a09aa314eb46231a295b5) switched from SCHEDULED to CANCELED.
2020-05-03 22:51:40,205 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window(GlobalWindows(), CountTrigger, CountEvictor, PindaoReduce, PassThroughWindowFunction) -> Sink: pdrdreduce (1/1) (f0258b8006ced3f5cdd787d454e8ddc8) switched from SCHEDULED to CANCELED.
2020-05-03 22:51:40,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or fail the job pindaoredian (a189c7f3d756b8745fb07c7124502dec) if no longer possible.
2020-05-03 22:51:40,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job pindaoredian (a189c7f3d756b8745fb07c7124502dec) switched from state FAILING to RESTARTING.
2020-05-03 22:51:40,217 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Restarting the job pindaoredian (a189c7f3d756b8745fb07c7124502dec).
2020-05-03 22:51:50,218 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job pindaoredian (a189c7f3d756b8745fb07c7124502dec) switched from state RESTARTING to CREATED.
2020-05-03 22:51:50,219 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job pindaoredian (a189c7f3d756b8745fb07c7124502dec) switched from state CREATED to RUNNING.
2020-05-03 22:51:50,219 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Map (1/1) (f1d8e3f1df3c25820190ba00bd1cbcbf) switched from CREATED to SCHEDULED.
2020-05-03 22:51:50,219 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window(GlobalWindows(), CountTrigger, CountEvictor, PindaoReduce, PassThroughWindowFunction) -> Sink: pdrdreduce (1/1) (1dd55c67f10b3cb2c0962f3bd2f20958) switched from CREATED to SCHEDULED.
2020-05-03 22:51:50,220 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPool - Requesting new slot [SlotRequestId{1f915c06c6ef004f5d2504065d517bc7}] and profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager.
2020-05-03 22:51:50,220 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job a189c7f3d756b8745fb07c7124502dec with allocation id AllocationID{da4e0ae01707e2cc6211e6f968ee163d}.
2020-05-03 22:51:53,036 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Map (1/1) of job a189c7f3d756b8745fb07c7124502dec is not in state RUNNING but SCHEDULED instead. Aborting checkpoint.
2020-05-03 22:51:58,036 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Map (1/1) of job a189c7f3d756b8745fb07c7124502dec is not in state RUNNING but SCHEDULED instead. Aborting checkpoint.
2020-05-03 22:52:03,036 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Map (1/1) of job a189c7f3d756b8745fb07c7124502dec is not in state RUNNING but SCHEDULED instead. Aborting checkpoint.
2020-05-03 22:52:08,036 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Map (1/1) of job a189c7f3d756b8745fb07c7124502dec is not in state RUNNING but SCHEDULED instead. Aborting checkpoint.
2020-05-03 22:52:13,036 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Map (1/1) of job a189c7f3d756b8745fb07c7124502dec is not in state RUNNING but SCHEDULED instead. Aborting checkpoint.
2020-05-03 22:52:18,036 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Map (1/1) of job a189c7f3d756b8745fb07c7124502dec is not in state RUNNING but SCHEDULED instead. Aborting checkpoint.
2020-05-03 22:52:23,036 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Map (1/1) of job a189c7f3d756b8745fb07c7124502dec is not in state RUNNING but SCHEDULED instead. Aborting checkpoint.
2020-05-03 22:52:28,036 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Map (1/1) of job a189c7f3d756b8745fb07c7124502dec is not in state RUNNING but SCHEDULED instead. Aborting checkpoint.
2020-05-03 22:52:33,036 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Map (1/1) of job a189c7f3d756b8745fb07c7124502dec is not in state RUNNING but SCHEDULED instead. Aborting checkpoint.
2020-05-03 22:52:38,036 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Map (1/1) of job a189c7f3d756b8745fb07c7124502dec is not in state RUNNING but SCHEDULED instead. Aborting checkpoint.
2020-05-03 22:52:43,036 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Map (1/1) of job a189c7f3d756b8745fb07c7124502dec is not in state RUNNING but SCHEDULED instead. Aborting checkpoint.
2020-05-03 22:52:48,036 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Map (1/1) of job a189c7f3d756b8745fb07c7124502dec is not in state RUNNING but SCHEDULED instead. Aborting checkpoint.
2020-05-03 22:52:53,036 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Map (1/1) of job a189c7f3d756b8745fb07c7124502dec is not in state RUNNING but SCHEDULED instead. Aborting checkpoint.
2020-05-03 22:52:58,036 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Map (1/1) of job a189c7f3d756b8745fb07c7124502dec is not in state RUNNING but SCHEDULED instead. Aborting checkpoint.
2020-05-03 22:53:03,036 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Map (1/1) of job a189c7f3d756b8745fb07c7124502dec is not in state RUNNING but SCHEDULED instead. Aborting checkpoint.
總結
今天大部分時間都用在了調通服務器上，接下來將編寫前端服務，如果有知道爲什麼Task Solts爲0的可以私信或者留言。
具體代碼可參照我的git項目地址,現有代碼均已通過測試可以使用，後續會持續更新，直到項目結束，不懂的細節，可以關注公衆號，後臺留言，會細緻解答。
git地址:https://github.com/jyqjyq/filnkDS.git
