LTS詳細使用流程

前言

工作需要使用實時任務,雖然該框架目前屬於無人維護的狀態,但綜合考量下還是使用LTS。編譯過程參考LTS框架,不再過多敘述,本文版本v1.7.2。

啓動流程

  • 基礎環境:這裏選用Zookeeper、MongoDB以及MySQL作爲前提條件進行啓動,具體不展開
  • 按照官方文檔編譯出dist,備份好原有conf(很重要)
  • 啓動lts-admin(含monitor):
    • 修改conf文件下的lts-admin.cfg,改爲自己的版本,這裏給個參考
      port=7777
      
      // 後臺的用戶名密碼
      console.username=admin
      console.password=admin
      
      # 註冊中心地址,可以是zk,也可以是redis
      registryAddress=zookeeper://127.0.0.1:2181
      # registryAddress=redis://127.0.0.1:6379
      
      # 集羣名稱
      clusterName=test_cluster
      
      # zk客戶端,可選值 zkclient, curator
      configs.zk.client=zkclient
      
      # ------ 這個是Admin存儲數據的地方,也可以和JobQueue的地址一樣 ------
      configs.jdbc.url=jdbc:mysql://127.0.0.1:3306/lts
      configs.jdbc.username=root
      configs.jdbc.password=123456
      
      # admin 數據使用mysql 默認 mysql, 可以自行擴展
      jdbc.datasource.provider=mysql
      
      # 使用 可選值  fastjson, jackson
      # configs.lts.json=fastjson
      
      # 是否在admin啓動monitor服務, monitor服務也可以單獨啓動
      lts.monitorAgent.enable=true
      
      #======================以下相關配置是JobTracker的JobQueue和JobLogger的相關配置 要保持和JobTracker一樣==========================
      ## (可選配置)jobT. 開頭的, 因爲JobTracker和Admin可能使用的數據庫不是同一個
      # LTS業務日誌, 可選值 mysql, mongo
      jobT.job.logger=mongo
      # ---------以下是任務隊列配置-----------
      # 任務隊列,可選值 mysql, mongo
      jobT.job.queue=mongo
      
      # ------ 1. 如果是mysql作爲任務隊列 (如果不配置,表示和Admin的在一個數據庫)------
      # jobT.jdbc.url=jdbc:mysql://127.0.0.1:3306/lts
      # jobT.jdbc.username=root
      # jobT.jdbc.password=123456
      
      # ------ 2. 如果是mongo作爲任務隊列 ------
      jobT.mongo.addresses=127.0.0.1:27017
      jobT.mongo.database=lts
      # jobT.mongo.username=xxx #如果有的話
      # jobT.mongo.password=xxx #如果有的話
      
      # admin 數據使用mysql 默認 mysql, 可以自行擴展
      # jobT.jdbc.datasource.provider=mysql

       

    • 同時修改lts-monitor.cfg(若上面monitor配置沒有設爲true則不用修改)
      
      # 註冊中心地址,可以是zk,也可以是redis
      registryAddress=zookeeper://127.0.0.1:2181
      # registryAddress=redis://127.0.0.1:6379
      
      # 集羣名稱
      clusterName=test_cluster
      
      # LTS業務日誌, 可選值 mysql, mongo
      configs.job.logger=mongo
      
      # zk客戶端,可選值 zkclient, curator
      configs.zk.client=zkclient
      
      # ---------以下是任務隊列配置-----------
      # 任務隊列,可選值 mysql, mongo
      configs.job.queue=mongo
      
      # ------ 1. 如果是mysql作爲任務隊列 ------
      configs.jdbc.url=jdbc:mysql://127.0.0.1:3306/lts
      configs.jdbc.username=root
      configs.jdbc.password=123456
      
      # ------ 2. 如果是mongo作爲任務隊列 ------
      configs.mongo.addresses=127.0.0.1:27017
      configs.mongo.database=lts
      # configs.mongo.username=xxx #如果有的話
      # configs.mongo.password=xxx #如果有的話
      
      # admin 數據使用mysql, h2 默認 h2 embedded
      jdbc.datasource.provider=mysql
      
      # 使用 可選值  fastjson, jackson
      # configs.lts.json=fastjson

       

    • 然後進入bin/lts-admin start啓動,注意留意有無錯誤,一般錯誤發生在外部環境連接不上或者任務隊列兩個之一沒有配好賬號密碼(雖然它只用一個隊列,但兩種方式都要配好,應該算是個bug)。成功後訪問admin地址,看到存在MONITOR節點即算成功
  • 啓動jobtracker
    • 修改conf/zoo/jobtracker.cfg,核心還是同上,兩個隊列都要寫
      
      # 註冊中心地址,可以是zk,也可以是redis
      registryAddress=zookeeper://127.0.0.1:2181
      
      # JobTracker的監聽端口
      listenPort=3502
      
      # 集羣名稱
      clusterName=test_cluster
      
      # LTS業務日誌, 可選值 console, mysql, mongo
      configs.job.logger=mongo
      
      # zk客戶端,可選值 zkclient, curator
      configs.zk.client=zkclient
      
      # ---------以下是任務隊列配置-----------
      # 任務隊列,可選值 mysql, mongo
      configs.job.queue=mongo
      
      # ------ 1. 如果是mysql作爲任務隊列 ------
      configs.jdbc.url=jdbc:mysql://127.0.0.1:3306/lts
      configs.jdbc.username=root
      configs.jdbc.password=123456
      
      # ------ 2. 如果是mongo作爲任務隊列 ------
      configs.mongo.addresses=127.0.0.1:27017
      configs.mongo.database=lts
      # configs.mongo.username=xxx #如果有的話
      # configs.mongo.password=xxx #如果有的話

       

    • 啓動bin/jobtracker zoo start,無錯誤的話刷新admin網頁,就看到第二個節點JOB_TRACKER出現了
  • 自定義tasktracker啓動
    • 參考POM
      <dependencies>
          <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <scope>test</scope>
          </dependency>
          
          <!-- https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter -->
      	<dependency>
      	    <groupId>org.springframework.boot</groupId>
      	    <artifactId>spring-boot-starter</artifactId>
          </dependency>
      
      	<!-- 熱部署 -->
          <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <optional>true</optional>
          </dependency>
      
      	<dependency>
      	    <groupId>com.github.ltsopensource</groupId>
      	    <artifactId>lts-spring</artifactId>
      	    <version>1.7.2-SNAPSHOT</version>
      	</dependency>
          
           <dependency>
      	    <groupId>com.github.ltsopensource</groupId>
      	    <artifactId>lts-core</artifactId>
      	    <version>1.7.2-SNAPSHOT</version>
      	</dependency>
      	
      	 <dependency>
      	    <groupId>com.github.ltsopensource</groupId>
      	    <artifactId>lts-tasktracker</artifactId>
      	    <version>1.7.2-SNAPSHOT</version>
      	</dependency>
      	
      	<dependency>
      	    <groupId>com.alibaba</groupId>
      	    <artifactId>fastjson</artifactId>
      	    <version>1.2.58</version>
      	</dependency>
      	
      	<!-- https://mvnrepository.com/artifact/io.netty/netty-all -->
      	<dependency>
      	    <groupId>io.netty</groupId>
      	    <artifactId>netty-all</artifactId>
      	</dependency>
      	
      	<!-- https://mvnrepository.com/artifact/org.iq80.leveldb/leveldb -->
      	<dependency>
      	    <groupId>org.iq80.leveldb</groupId>
      	    <artifactId>leveldb</artifactId>
      	    <version>0.11</version>
      	</dependency>
      	
      	<!-- https://mvnrepository.com/artifact/org.fusesource.leveldbjni/leveldbjni-all -->
      	<dependency>
      	    <groupId>org.fusesource.leveldbjni</groupId>
      	    <artifactId>leveldbjni-all</artifactId>
      	    <version>1.8</version>
      	</dependency>
      
      	<!-- https://mvnrepository.com/artifact/com.101tec/zkclient -->
      	<dependency>
      	    <groupId>com.101tec</groupId>
      	    <artifactId>zkclient</artifactId>
      	    <version>0.11</version>
      	</dependency>
      	
      	<!-- https://mvnrepository.com/artifact/org.apache.zookeeper/zookeeper -->
      	<dependency>
      	    <groupId>org.apache.zookeeper</groupId>
      	    <artifactId>zookeeper</artifactId>
      	    <version>3.4.14</version>
      	    <type>pom</type>
      	</dependency>
        </dependencies>
        
        <dependencyManagement>
          <dependencies>
            <!--支持Spring Boot 2.1.X-->
            <dependency>
              <groupId>org.springframework.boot</groupId>
              <artifactId>spring-boot-dependencies</artifactId>
              <version>2.1.4.RELEASE</version>
              <type>pom</type>
              <scope>import</scope>
            </dependency>
          </dependencies>
        </dependencyManagement>

       

    • 參考配置文件
      lts:
        tasktracker:
          cluster-name: test_cluster
          registry-address: zookeeper://127.0.0.1:2181
          work-threads: 32
          node-group: test_TaskTracker
          dispatch-runner:
            enable: true
            #shard-value: taskId

       

    • 如果在Application啓動@EnableTaskTracker的話就會使用默認的TaskTracker,目前會導致使用帶上@JobRunner4TaskTracker裝飾的自定義JobRunner出現Can not find JobRunner by Shard Value的bug
    • 在這裏使用自定義Config來註冊一個TaskTracker,參考代碼
      @Configuration
      public class Config implements ApplicationContextAware {
      	private ApplicationContext applicationContext;
      	
      	@Value("${lts.tasktracker.cluster-name}")
      	private String clusterName;
      	
      	@Value("${lts.tasktracker.node-group}")
      	private String nodeGroup;
      	
      	@Value("${lts.tasktracker.work-threads}")
      	private Integer workThreads;
      	
      	@Value("${lts.tasktracker.registry-address}")
      	private String address;
      
          @Override
          public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
              this.applicationContext = applicationContext;
          }
      
          @Bean(name = "taskTracker")
          public TaskTracker getTaskTracker() throws Exception {
              TaskTrackerAnnotationFactoryBean factoryBean = new TaskTrackerAnnotationFactoryBean();
              factoryBean.setApplicationContext(applicationContext);
              factoryBean.setClusterName(clusterName);
              factoryBean.setJobRunnerClass(JobRunnerDispatcher.class);
              factoryBean.setWorkThreads(workThreads);
              factoryBean.setShardField("taskId"); // 待定
              factoryBean.setNodeGroup(nodeGroup);
              factoryBean.setBizLoggerLevel("INFO");
              factoryBean.setRegistryAddress(address);
      
              factoryBean.afterPropertiesSet();
              factoryBean.start();
              return factoryBean.getObject();
          }
      }

       

    • 用Application啓動,無錯誤的話,會在admin網頁看到一個TASK_TRACKER節點上線
  • 啓動JobClient
    • 參考POM
      <dependencies>
          <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <scope>test</scope>
          </dependency>
          
          <!-- https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter -->
      	<dependency>
      	    <groupId>org.springframework.boot</groupId>
      	    <artifactId>spring-boot-starter</artifactId>
          </dependency>
      
      	<!-- 熱部署 -->
          <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <optional>true</optional>
          </dependency>
          
          <dependency>
      	  <groupId>org.springframework.boot</groupId>
      	  <artifactId>spring-boot-starter-web</artifactId>
      	</dependency>
          
           <dependency>
      	    <groupId>com.github.ltsopensource</groupId>
      	    <artifactId>lts-core</artifactId>
      	    <version>1.7.2-SNAPSHOT</version>
      	</dependency>
      	
      	<dependency>
      	    <groupId>com.github.ltsopensource</groupId>
      	    <artifactId>lts-spring</artifactId>
      	    <version>1.7.2-SNAPSHOT</version>
      	</dependency>
      	
      	 <dependency>
      	    <groupId>com.github.ltsopensource</groupId>
      	    <artifactId>lts-jobclient</artifactId>
      	    <version>1.7.2-SNAPSHOT</version>
      	</dependency>
      	
      	<dependency>
      	    <groupId>com.alibaba</groupId>
      	    <artifactId>fastjson</artifactId>
      	    <version>1.2.58</version>
      	</dependency>
      	
      	<!-- https://mvnrepository.com/artifact/io.netty/netty-all -->
      	<dependency>
      	    <groupId>io.netty</groupId>
      	    <artifactId>netty-all</artifactId>
      	</dependency>
      	
      	<!-- https://mvnrepository.com/artifact/org.iq80.leveldb/leveldb -->
      	<dependency>
      	    <groupId>org.iq80.leveldb</groupId>
      	    <artifactId>leveldb</artifactId>
      	    <version>0.11</version>
      	</dependency>
      	
      	<!-- https://mvnrepository.com/artifact/org.fusesource.leveldbjni/leveldbjni-all -->
      	<dependency>
      	    <groupId>org.fusesource.leveldbjni</groupId>
      	    <artifactId>leveldbjni-all</artifactId>
      	    <version>1.8</version>
      	</dependency>
      
      	<!-- https://mvnrepository.com/artifact/com.101tec/zkclient -->
      	<dependency>
      	    <groupId>com.101tec</groupId>
      	    <artifactId>zkclient</artifactId>
      	    <version>0.11</version>
      	</dependency>
      	
      	<!-- https://mvnrepository.com/artifact/org.apache.zookeeper/zookeeper -->
      	<dependency>
      	    <groupId>org.apache.zookeeper</groupId>
      	    <artifactId>zookeeper</artifactId>
      	    <version>3.4.14</version>
      	    <type>pom</type>
      	</dependency>
      	
        </dependencies>
      	
        <dependencyManagement>
          <dependencies>
            <!--支持Spring Boot 2.1.X-->
            <dependency>
              <groupId>org.springframework.boot</groupId>
              <artifactId>spring-boot-dependencies</artifactId>
              <version>2.1.4.RELEASE</version>
              <type>pom</type>
              <scope>import</scope>
            </dependency>
          </dependencies>
        </dependencyManagement>

       

    • 參考配置文件
      lts:
        jobclient:
          cluster-name: test_cluster
          registry-address: zookeeper://127.0.0.1:2181
          node-group: test_jobClient
          use-retry-client: true

       

    • Application加上@EnableJobClient裝飾
    • (可選)任務完成處理代碼
      @JobCompletedHandler4JobClient
      public class JobCompletedHandlerImpl implements JobCompletedHandler {
      
      	@Override
      	public void onComplete(List<JobResult> jobResults) {
      		 // 任務執行反饋結果處理
              if (CollectionUtils.isNotEmpty(jobResults)) {
                  for (JobResult jobResult : jobResults) {
                      System.out.println(new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(new Date()) + " 任務執行完成:" + jobResult);
                  }
              }
      	}
      
      }

       

    • 啓動後無錯誤可以看到JOB_CLIENT節點上線

BUG

目前admin網頁裏,打開任務日誌查詢不到數據,但是在MongoDB裏是可以看到日誌數據的,待解決

 

 

 

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章