quartz + kettle8二次開發-實現集羣高可用

文章目錄

效果演示

背景說明

1、etl是報表開發/大數據開發的必備步驟，而其中免費、開源、且最好用的工具就是kettle。關於kettle（pdi-ce）可視化界面工具的使用和介紹請大家自行百度。本文主要介紹如何把kettle集成到我們的項目中，來進行二次開發。
2、通過查詢資料發現：大多數公司的做法是部署一套kettle集羣（僞集羣），然後使用jenkins去定時調度kettle集羣。甚至有公司直接用linux中的cron來做定時任務調度的工作。但是這裏存在兩個問題：
- 第一：kettle集羣爲“主-從”式的僞集羣，存在單點故障問題，如下圖：
  
  （當然網上也有很多解決辦法，例如，master可以使用keeplive做成雙活來避免單點問題，但是需要多引入一個服務。增加維護成本）
- 第二：定時任務本身也存在單點故障問題。
  
  （這個問題網上也有解決辦法，例如使用第三方開源的任務調度系統，例如kettle-manager等，其實km的底層調度仍然用的quartz，也就是本文準備介紹的。km擁有界面化操作，能夠直接在界面上對定時job進行crud，個人覺得還是很方便的。感興趣的可以自行研究。）
3、本文介紹的quartz+kettle架構
**本項目的做法是：**直接把kettle集成到程序中，然後使用quartz來調度kettle。不再需要依賴已有的kettle集羣，並且支持服務的高可用。

（注意：由於不再依賴kettle集羣，故沒有master/slave的區別，也沒有master分發任務的動作。所有任務在同一時刻只會在一個實例上做，會造成實例壓力大的問題。處理任務的效率可能會低於僞集羣式）

項目結構

開發步驟

項目的核心邏輯是：
1、在springboot項目啓動時，會最先執行InitRunner類，這個類會把quartz的定時任務寫入到數據庫中；
2、當定時時間到時，會去執行Job類，這個類會去調用kettle的方法，從而執行kettle腳本。
項目開發步驟是：

1、引入quartz依賴，和kettle依賴

注意其中的kettle三個核心庫

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.1.6.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.wllfengshu</groupId>
    <artifactId>etl-kettle</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>etl-kettle</name>
    <description>Demo project for Spring Boot</description>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
        <kettle.version>8.2.0.0-342</kettle.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-aop</artifactId>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.45</version>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.54</version>
        </dependency>
        <dependency>
            <groupId>net.logstash.logback</groupId>
            <artifactId>logstash-logback-encoder</artifactId>
            <version>5.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-vfs2</artifactId>
            <version>2.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>3.8.1</version>
        </dependency>
        <dependency>
            <groupId>commons-lang</groupId>
            <artifactId>commons-lang</artifactId>
            <version>2.6</version>
        </dependency>
        <dependency>
            <groupId>commons-codec</groupId>
            <artifactId>commons-codec</artifactId>
            <version>1.11</version>
        </dependency>
        <dependency>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>2.6</version>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>1.18.4</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-quartz</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-batch</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-thymeleaf</artifactId>
        </dependency>
        <dependency>
            <groupId>org.mybatis.spring.boot</groupId>
            <artifactId>mybatis-spring-boot-starter</artifactId>
            <version>1.3.2</version>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>druid-spring-boot-starter</artifactId>
            <version>1.1.10</version>
        </dependency>
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>20.0</version>
        </dependency>
        <!--kettle start-->
        <dependency>
            <groupId>pentaho-kettle</groupId>
            <artifactId>kettle-core</artifactId>
            <version>${kettle.version}</version>
        </dependency>
        <dependency>
            <groupId>pentaho-kettle</groupId>
            <artifactId>kettle-engine</artifactId>
            <version>${kettle.version}</version>
        </dependency>
        <dependency>
            <groupId>pentaho-kettle</groupId>
            <artifactId>metastore</artifactId>
            <version>${kettle.version}</version>
        </dependency>
        <!--kettle end-->
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <configuration>
                    <archive>
                        <manifest>
                            <addClasspath>true</addClasspath>
                            <useUniqueVersions>false</useUniqueVersions>
                            <classpathPrefix>lib/</classpathPrefix>
                            <mainClass>com.wllfengshu.etlkettle.EtlKettleApplication</mainClass>
                        </manifest>
                    </archive>
                    <excludes>
                        <exclude>/workspace/</exclude>
                    </excludes>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

2、配置quartz

server:
  port: 8080
  servlet:
    context-path: /etl/1.0
spring:
  application:
    name: etl
  datasource:
    url: ${db_url}
    username: ${db_username}
    password: ${db_password}
    driver-class-name: com.mysql.jdbc.Driver
    type: com.alibaba.druid.pool.DruidDataSource
  quartz:
    job-store-type: jdbc
    jdbc:
      initialize-schema: always
    properties:
      org:
        quartz:
          scheduler:
            instanceName: QuartzScheduler
            instanceId: CLUSTERED
          jobStore:
            class: org.quartz.impl.jdbcjobstore.JobStoreTX
            driverDelegateClass: org.quartz.impl.jdbcjobstore.StdJDBCDelegate
            tablePrefix: ETL_QRTZ_
            isClustered: true
            clusterCheckinInterval: 10000
            useProperties: false
          threadPool:
            class: org.quartz.simpl.SimpleThreadPool
            threadCount: 50
            threadPriority: 5
            threadsInheritContextClassLoaderOfInitializingThread: true
logging:
  level:
    cc.wellcloud.etl.dao : debug

3、編寫系統初始化類，在程序啓動時，添加quartz定時任務

package com.wllfengshu.etlkettle.init;

import com.wllfengshu.etlkettle.common.Constant;
import com.wllfengshu.etlkettle.model.QuartzJob;
import com.wllfengshu.etlkettle.utils.QuartzUtil;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.CommandLineRunner;
import org.springframework.stereotype.Component;

/**
 * 系統初始化
 * @author wangll
 */
@Slf4j
@Component
@RequiredArgsConstructor
public class InitRunner implements CommandLineRunner {

    private static final String CRON_MINUTE = "* */2 * * * ?";
    private static final String JOB_NAME = "etl_minute";
    private static final String TRIGGER_NAME = "etl_minute";

    @NonNull
    private QuartzUtil quartzUtil;

    @Override
    public void run(String... args){
        log.info("正在初始化全部定時任務...");
        QuartzJob m = new QuartzJob();
        m.setJobName(JOB_NAME);
        m.setTriggerName(TRIGGER_NAME);
        m.setCronExpression(CRON_MINUTE);
        m.setDescription("etl 2分鐘的定時任務");
        quartzUtil.addJob(m,Constant.JOB_CLASS_NAME);
        log.info("定時任務添加完畢");
    }
}

4、編寫quartz任務執行類，這個類中會去調用kettle的方法，從而執行kettle腳本

package com.wllfengshu.etlkettle.job;

import com.wllfengshu.etlkettle.common.Constant;
import com.wllfengshu.etlkettle.utils.DbUtil;
import com.wllfengshu.etlkettle.utils.KettleUtil;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.quartz.JobExecutionContext;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.scheduling.quartz.QuartzJobBean;
import org.springframework.stereotype.Component;

import java.io.File;
import java.util.HashMap;
import java.util.Map;

/**
 * 定時任務
 * @author wangll
 */
@Slf4j
@Component
@RequiredArgsConstructor
public class Job extends QuartzJobBean {

    @NonNull
    private KettleUtil kettleUtil;

    @Value("${db_url}")
    private String dbUrl;
    @Value("${db_username}")
    private String dbUsername;
    @Value("${db_password}")
    private String dbPassword;

    @Override
    protected void executeInternal(JobExecutionContext jobExecutionContext){
        //1 設置kettle參數(把程序中的參數傳入到kettle中)
        Map<String,String> params = new HashMap<>(16);
        //1.1 設置數據庫信息
        Map<String, String> mapInput = DbUtil.dealDb(dbUrl, dbUsername, dbPassword);
        params.put("dbIp",mapInput.get("dbIp"));
        params.put("dbPort",mapInput.get("dbPort"));
        params.put("dbName",mapInput.get("dbName"));
        params.put("dbUsername",mapInput.get("dbUsername"));
        params.put("dbPassword",mapInput.get("dbPassword"));
        log.info("kettle配置的參數：{}",params);
        //2 取所有kettle腳本
        File workspace = new File(Constant.WORKSPACE_PATH);
        File[] files = workspace.listFiles();
        //2.1 執行kettle
        for (File file:files){
            String fileName = file.getName();
            try {
                if (fileName.endsWith(Constant.FILE_SUFFIX_KTR)){
                    kettleUtil.runTrans(file, params);
                }else if (fileName.endsWith(Constant.FILE_SUFFIX_KJB)){
                    kettleUtil.runJob(file, params);
                }else {
                    log.error("文件名不合法,它必須是ktr或kjb");
                    continue;
                }
            } catch (Exception e) {
                log.error("執行ktr或kjb失敗,fileName={}",fileName,e);
            }
        }
    }
}

5、其他類

//(1) 常量類
package com.wllfengshu.etlkettle.common;

import lombok.extern.slf4j.Slf4j;

/**
 * 公共常量集合
 *
 * @author wangll
 */
@Slf4j
public class Constant {

    /**
     * ktr文件後綴
     */
    public static final String FILE_SUFFIX_KTR = "ktr";

    /**
     * kjb文件後綴
     */
    public static final String FILE_SUFFIX_KJB = "kjb";

    /**
     * job組
     */
    public static final String JOB_GROUP = "ETL";

    /**
     * job的執行類
     */
    public static final String JOB_CLASS_NAME = "com.wllfengshu.etlkettle.job.Job";

    /**
     * ktr和kjb文件存放路徑
     */
    public static final String WORKSPACE_PATH = "/home/listen/Apps/workspace";

}

//(2) quartz的實體類
package com.wllfengshu.etlkettle.model;

import lombok.Data;

import java.io.Serializable;

/**
 * job實體類
 * @author wangll
 */
@Data
public class QuartzJob implements Serializable {

    private static final long serialVersionUID = 1L;

    /**
     * 任務名稱
     */
    private String jobName;

    /**
     * 任務分組
     */
    private String jobGroup;

    /**
     * 任務描述
     */
    private String description;

    /**
     * 執行類
     */
    private String jobClassName;

    /**
     * 執行時間
     */
    private String cronExpression;

    /**
     * 觸發器名
     */
    private String triggerName;

    /**
     * 觸發器狀態
     */
    private String triggerState;

}

//(3) kettle執行類
package com.wllfengshu.etlkettle.utils;

import lombok.NonNull;
import lombok.extern.slf4j.Slf4j;
import org.pentaho.di.core.KettleEnvironment;
import org.pentaho.di.core.exception.KettleException;
import org.pentaho.di.core.util.EnvUtil;
import org.pentaho.di.job.Job;
import org.pentaho.di.job.JobMeta;
import org.pentaho.di.trans.Trans;
import org.pentaho.di.trans.TransMeta;
import org.springframework.stereotype.Component;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.util.Map;

/**
 * kettle工具類
 * 使用靜態塊的原因：
 *    1.執行ktr和kjb都要執行初始化，爲了代碼的複用；
 *    2.由於quartz的定時任務是異步的，在springBoot剛剛啓動完成就可能會執行ktr和kjb，所以kettle的初始化工作
 *      必須在springBoot啓動完成前執行，故使用靜態塊；
 * @author wangll
 */
@Slf4j
@Component
public class KettleUtil {

    static {
        log.info("正在初始化kettle ...");
        try {
            KettleEnvironment.init();
            EnvUtil.environmentInit();
        }catch (Exception e){
            log.error("初始化kettle失敗",e);
            System.exit(0);
        }
        log.info("初始化kettle成功！");
    }

    /**
     * 執行ktr文件
     * @param file
     * @param params
     */
    public void runTrans(@NonNull File file, Map<String,String> params) throws KettleException, FileNotFoundException {
        log.info("正在執行ktr ...");
        TransMeta transMeta = new TransMeta(new FileInputStream(file), null, true, null, null);
        Trans trans = new Trans(transMeta);
        params.forEach((k,v)->
                trans.setVariable(k,v)
        );
        trans.prepareExecution(null);
        trans.startThreads();
        trans.waitUntilFinished();
        if (trans.getErrors() != 0) {
            log.error("執行ktr過程中存在錯誤");
        }
        log.info("ktr執行完畢！");
    }

    /**
     * 執行kjb文件
     * @param file
     * @param params
     */
    public void runJob(@NonNull File file,Map<String,String> params) throws KettleException, FileNotFoundException {
        log.info("正在執行kjb ...");
        JobMeta jobMeta = new JobMeta(new FileInputStream(file), null, null);
        Job job = new Job(null, jobMeta);
        params.forEach((k,v)->
                job.setVariable(k,v)
        );
        job.start();
        job.waitUntilFinished();
        if (job.getErrors() != 0) {
            log.error("執行kjb過程中存在錯誤");
        }
        log.info("kjb執行完畢...");
    }
}


//(4) quartz工具類
package com.wllfengshu.etlkettle.utils;

import com.wllfengshu.etlkettle.common.Constant;
import com.wllfengshu.etlkettle.model.QuartzJob;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.quartz.*;
import org.springframework.stereotype.Component;

/**
 * quartz工具類
 * @author wangll
 */
@Slf4j
@Component
@RequiredArgsConstructor
public class QuartzUtil {

    @NonNull
    private Scheduler scheduler;

    /**
     * 添加任務
     */
    public void addJob(@NonNull QuartzJob quartz, @NonNull String jobClassName){
        log.info("正在添加定時任務，quartz={} ...",quartz);
        try {
            //1 構建jobKey判斷任務是否已經存在
            JobKey jobKey = new JobKey(quartz.getJobName(),Constant.JOB_GROUP);
            if (scheduler.checkExists(jobKey)) {
                log.info("該定時任務已存在！");
                return;
            }
            //2 構建job信息
            Class cls = Class.forName(jobClassName);
            JobDetail job = JobBuilder.newJob(cls).withIdentity(jobKey)
                    .withDescription(quartz.getDescription()).build();
            //3 觸發時間點
            CronScheduleBuilder cronScheduleBuilder = CronScheduleBuilder.cronSchedule(quartz.getCronExpression().trim());
            //4 構建triggerKey判斷觸發器是否已經存在
            TriggerKey triggerKey = new TriggerKey(quartz.getTriggerName(),Constant.JOB_GROUP);
            if (scheduler.checkExists(triggerKey)) {
                log.info("該觸發器已存在！");
                return;
            }
            Trigger trigger = TriggerBuilder.newTrigger().withIdentity(triggerKey)
                    .startNow().withSchedule(cronScheduleBuilder).build();
            //5 交由Scheduler安排觸發
            scheduler.scheduleJob(job, trigger);
        } catch (Exception e) {
            log.error("保存job失敗",e);
            // throw CustomException
        }
        log.info("定時任務添加成功！");
    }
}

//(5) 把連接mysql完整的url分解爲ip/port/dbName
package com.wllfengshu.etlkettle.utils;

import lombok.extern.slf4j.Slf4j;

import java.util.HashMap;
import java.util.Map;

/**
 * 數據庫工具類
 * @author wangll
 */
@Slf4j
public class DbUtil {

    /**
     * 處理數據庫配置
     * @param dbUrl
     * @param dbUsername
     * @param dbPassword
     * @return
     */
    public static Map<String,String> dealDb(String dbUrl, String dbUsername, String dbPassword) {
        Map<String,String> map = new HashMap<>(5);
        //1 截取出ip:port/dbName
        String temp = dbUrl.substring("jdbc:mysql://".length(),
                dbUrl.contains("?") ? dbUrl.indexOf("?") : dbUrl.length());
        //2 分離ip
        String[] ipPortDb = temp.split(":");
        map.put("dbIp",ipPortDb[0]);
        //3 分離port和dbName
        String[] portDb = ipPortDb[1].split("/");
        map.put("dbPort",portDb[0]);
        map.put("dbName",portDb[1]);
        //4 設置用戶名和密碼
        map.put("dbUsername",dbUsername);
        map.put("dbPassword",dbPassword);
        log.info("數據庫信息處理完畢，dbInfo={}",map);
        return map;
    }
}

//(6)  springboot啓動類
package com.wllfengshu.etlkettle;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class EtlKettleApplication {

    public static void main(String[] args) {
        SpringApplication.run(EtlKettleApplication.class, args);
    }

}

注意：

本項目使用了lombok插件，請先確保idea安裝了該插件；
本文是把數據庫信息作爲參數傳到kettle腳本中的，其實kettle庫中的DatabaseMeta類可以直接設置數據庫信息，大家可以自己優化；
本項目是把kettle腳本（ktr和kjb）存放在項目的resources/workspace目錄中，然後在docker打包時把所有kettle腳本都移動到/home/listen/Apps/workspace路徑下。大家也可以直接把腳本和程序打包到一個jar中，但是需要解決如何讀取jar中的文件的問題。

效果演示

1、quartz會自動建表，表如下：

2、t_user表如下

3、使用腳本插入的效果：

4、把編寫的kettle腳本放到workspace目錄下，例如，insert_user.ktr腳本是：
（作用是：向數據庫的user表插入一條記錄）

<?xml version="1.0" encoding="UTF-8"?>
<transformation>
  <info>
    <name>插入user</name>
    <description/>
    <extended_description/>
    <trans_version/>
    <trans_type>Normal</trans_type>
    <trans_status>2</trans_status>
    <directory>/</directory>
    <parameters>
    </parameters>
    <log>
      <trans-log-table>
        <connection/>
        <schema/>
        <table/>
        <size_limit_lines/>
        <interval/>
        <timeout_days/>
        <field>
          <id>ID_BATCH</id>
          <enabled>Y</enabled>
          <name>ID_BATCH</name>
        </field>
        <field>
          <id>CHANNEL_ID</id>
          <enabled>Y</enabled>
          <name>CHANNEL_ID</name>
        </field>
        <field>
          <id>TRANSNAME</id>
          <enabled>Y</enabled>
          <name>TRANSNAME</name>
        </field>
        <field>
          <id>STATUS</id>
          <enabled>Y</enabled>
          <name>STATUS</name>
        </field>
        <field>
          <id>LINES_READ</id>
          <enabled>Y</enabled>
          <name>LINES_READ</name>
          <subject/>
        </field>
        <field>
          <id>LINES_WRITTEN</id>
          <enabled>Y</enabled>
          <name>LINES_WRITTEN</name>
          <subject/>
        </field>
        <field>
          <id>LINES_UPDATED</id>
          <enabled>Y</enabled>
          <name>LINES_UPDATED</name>
          <subject/>
        </field>
        <field>
          <id>LINES_INPUT</id>
          <enabled>Y</enabled>
          <name>LINES_INPUT</name>
          <subject/>
        </field>
        <field>
          <id>LINES_OUTPUT</id>
          <enabled>Y</enabled>
          <name>LINES_OUTPUT</name>
          <subject/>
        </field>
        <field>
          <id>LINES_REJECTED</id>
          <enabled>Y</enabled>
          <name>LINES_REJECTED</name>
          <subject/>
        </field>
        <field>
          <id>ERRORS</id>
          <enabled>Y</enabled>
          <name>ERRORS</name>
        </field>
        <field>
          <id>STARTDATE</id>
          <enabled>Y</enabled>
          <name>STARTDATE</name>
        </field>
        <field>
          <id>ENDDATE</id>
          <enabled>Y</enabled>
          <name>ENDDATE</name>
        </field>
        <field>
          <id>LOGDATE</id>
          <enabled>Y</enabled>
          <name>LOGDATE</name>
        </field>
        <field>
          <id>DEPDATE</id>
          <enabled>Y</enabled>
          <name>DEPDATE</name>
        </field>
        <field>
          <id>REPLAYDATE</id>
          <enabled>Y</enabled>
          <name>REPLAYDATE</name>
        </field>
        <field>
          <id>LOG_FIELD</id>
          <enabled>Y</enabled>
          <name>LOG_FIELD</name>
        </field>
        <field>
          <id>EXECUTING_SERVER</id>
          <enabled>N</enabled>
          <name>EXECUTING_SERVER</name>
        </field>
        <field>
          <id>EXECUTING_USER</id>
          <enabled>N</enabled>
          <name>EXECUTING_USER</name>
        </field>
        <field>
          <id>CLIENT</id>
          <enabled>N</enabled>
          <name>CLIENT</name>
        </field>
      </trans-log-table>
      <perf-log-table>
        <connection/>
        <schema/>
        <table/>
        <interval/>
        <timeout_days/>
        <field>
          <id>ID_BATCH</id>
          <enabled>Y</enabled>
          <name>ID_BATCH</name>
        </field>
        <field>
          <id>SEQ_NR</id>
          <enabled>Y</enabled>
          <name>SEQ_NR</name>
        </field>
        <field>
          <id>LOGDATE</id>
          <enabled>Y</enabled>
          <name>LOGDATE</name>
        </field>
        <field>
          <id>TRANSNAME</id>
          <enabled>Y</enabled>
          <name>TRANSNAME</name>
        </field>
        <field>
          <id>STEPNAME</id>
          <enabled>Y</enabled>
          <name>STEPNAME</name>
        </field>
        <field>
          <id>STEP_COPY</id>
          <enabled>Y</enabled>
          <name>STEP_COPY</name>
        </field>
        <field>
          <id>LINES_READ</id>
          <enabled>Y</enabled>
          <name>LINES_READ</name>
        </field>
        <field>
          <id>LINES_WRITTEN</id>
          <enabled>Y</enabled>
          <name>LINES_WRITTEN</name>
        </field>
        <field>
          <id>LINES_UPDATED</id>
          <enabled>Y</enabled>
          <name>LINES_UPDATED</name>
        </field>
        <field>
          <id>LINES_INPUT</id>
          <enabled>Y</enabled>
          <name>LINES_INPUT</name>
        </field>
        <field>
          <id>LINES_OUTPUT</id>
          <enabled>Y</enabled>
          <name>LINES_OUTPUT</name>
        </field>
        <field>
          <id>LINES_REJECTED</id>
          <enabled>Y</enabled>
          <name>LINES_REJECTED</name>
        </field>
        <field>
          <id>ERRORS</id>
          <enabled>Y</enabled>
          <name>ERRORS</name>
        </field>
        <field>
          <id>INPUT_BUFFER_ROWS</id>
          <enabled>Y</enabled>
          <name>INPUT_BUFFER_ROWS</name>
        </field>
        <field>
          <id>OUTPUT_BUFFER_ROWS</id>
          <enabled>Y</enabled>
          <name>OUTPUT_BUFFER_ROWS</name>
        </field>
      </perf-log-table>
      <channel-log-table>
        <connection/>
        <schema/>
        <table/>
        <timeout_days/>
        <field>
          <id>ID_BATCH</id>
          <enabled>Y</enabled>
          <name>ID_BATCH</name>
        </field>
        <field>
          <id>CHANNEL_ID</id>
          <enabled>Y</enabled>
          <name>CHANNEL_ID</name>
        </field>
        <field>
          <id>LOG_DATE</id>
          <enabled>Y</enabled>
          <name>LOG_DATE</name>
        </field>
        <field>
          <id>LOGGING_OBJECT_TYPE</id>
          <enabled>Y</enabled>
          <name>LOGGING_OBJECT_TYPE</name>
        </field>
        <field>
          <id>OBJECT_NAME</id>
          <enabled>Y</enabled>
          <name>OBJECT_NAME</name>
        </field>
        <field>
          <id>OBJECT_COPY</id>
          <enabled>Y</enabled>
          <name>OBJECT_COPY</name>
        </field>
        <field>
          <id>REPOSITORY_DIRECTORY</id>
          <enabled>Y</enabled>
          <name>REPOSITORY_DIRECTORY</name>
        </field>
        <field>
          <id>FILENAME</id>
          <enabled>Y</enabled>
          <name>FILENAME</name>
        </field>
        <field>
          <id>OBJECT_ID</id>
          <enabled>Y</enabled>
          <name>OBJECT_ID</name>
        </field>
        <field>
          <id>OBJECT_REVISION</id>
          <enabled>Y</enabled>
          <name>OBJECT_REVISION</name>
        </field>
        <field>
          <id>PARENT_CHANNEL_ID</id>
          <enabled>Y</enabled>
          <name>PARENT_CHANNEL_ID</name>
        </field>
        <field>
          <id>ROOT_CHANNEL_ID</id>
          <enabled>Y</enabled>
          <name>ROOT_CHANNEL_ID</name>
        </field>
      </channel-log-table>
      <step-log-table>
        <connection/>
        <schema/>
        <table/>
        <timeout_days/>
        <field>
          <id>ID_BATCH</id>
          <enabled>Y</enabled>
          <name>ID_BATCH</name>
        </field>
        <field>
          <id>CHANNEL_ID</id>
          <enabled>Y</enabled>
          <name>CHANNEL_ID</name>
        </field>
        <field>
          <id>LOG_DATE</id>
          <enabled>Y</enabled>
          <name>LOG_DATE</name>
        </field>
        <field>
          <id>TRANSNAME</id>
          <enabled>Y</enabled>
          <name>TRANSNAME</name>
        </field>
        <field>
          <id>STEPNAME</id>
          <enabled>Y</enabled>
          <name>STEPNAME</name>
        </field>
        <field>
          <id>STEP_COPY</id>
          <enabled>Y</enabled>
          <name>STEP_COPY</name>
        </field>
        <field>
          <id>LINES_READ</id>
          <enabled>Y</enabled>
          <name>LINES_READ</name>
        </field>
        <field>
          <id>LINES_WRITTEN</id>
          <enabled>Y</enabled>
          <name>LINES_WRITTEN</name>
        </field>
        <field>
          <id>LINES_UPDATED</id>
          <enabled>Y</enabled>
          <name>LINES_UPDATED</name>
        </field>
        <field>
          <id>LINES_INPUT</id>
          <enabled>Y</enabled>
          <name>LINES_INPUT</name>
        </field>
        <field>
          <id>LINES_OUTPUT</id>
          <enabled>Y</enabled>
          <name>LINES_OUTPUT</name>
        </field>
        <field>
          <id>LINES_REJECTED</id>
          <enabled>Y</enabled>
          <name>LINES_REJECTED</name>
        </field>
        <field>
          <id>ERRORS</id>
          <enabled>Y</enabled>
          <name>ERRORS</name>
        </field>
        <field>
          <id>LOG_FIELD</id>
          <enabled>N</enabled>
          <name>LOG_FIELD</name>
        </field>
      </step-log-table>
      <metrics-log-table>
        <connection/>
        <schema/>
        <table/>
        <timeout_days/>
        <field>
          <id>ID_BATCH</id>
          <enabled>Y</enabled>
          <name>ID_BATCH</name>
        </field>
        <field>
          <id>CHANNEL_ID</id>
          <enabled>Y</enabled>
          <name>CHANNEL_ID</name>
        </field>
        <field>
          <id>LOG_DATE</id>
          <enabled>Y</enabled>
          <name>LOG_DATE</name>
        </field>
        <field>
          <id>METRICS_DATE</id>
          <enabled>Y</enabled>
          <name>METRICS_DATE</name>
        </field>
        <field>
          <id>METRICS_CODE</id>
          <enabled>Y</enabled>
          <name>METRICS_CODE</name>
        </field>
        <field>
          <id>METRICS_DESCRIPTION</id>
          <enabled>Y</enabled>
          <name>METRICS_DESCRIPTION</name>
        </field>
        <field>
          <id>METRICS_SUBJECT</id>
          <enabled>Y</enabled>
          <name>METRICS_SUBJECT</name>
        </field>
        <field>
          <id>METRICS_TYPE</id>
          <enabled>Y</enabled>
          <name>METRICS_TYPE</name>
        </field>
        <field>
          <id>METRICS_VALUE</id>
          <enabled>Y</enabled>
          <name>METRICS_VALUE</name>
        </field>
      </metrics-log-table>
    </log>
    <maxdate>
      <connection/>
      <table/>
      <field/>
      <offset>0.0</offset>
      <maxdiff>0.0</maxdiff>
    </maxdate>
    <size_rowset>10000</size_rowset>
    <sleep_time_empty>50</sleep_time_empty>
    <sleep_time_full>50</sleep_time_full>
    <unique_connections>N</unique_connections>
    <feedback_shown>Y</feedback_shown>
    <feedback_size>50000</feedback_size>
    <using_thread_priorities>Y</using_thread_priorities>
    <shared_objects_file/>
    <capture_step_performance>N</capture_step_performance>
    <step_performance_capturing_delay>1000</step_performance_capturing_delay>
    <step_performance_capturing_size_limit>100</step_performance_capturing_size_limit>
    <dependencies>
    </dependencies>
    <partitionschemas>
    </partitionschemas>
    <slaveservers>
    </slaveservers>
    <clusterschemas>
    </clusterschemas>
    <created_user>-</created_user>
    <created_date>2019/07/11 16:30:05.348</created_date>
    <modified_user>-</modified_user>
    <modified_date>2019/07/11 16:30:05.348</modified_date>
    <key_for_session_key>H4sIAAAAAAAAAAMAAAAAAAAAAAA=</key_for_session_key>
    <is_key_private>N</is_key_private>
  </info>
  <notepads>
  </notepads>
  <connection>
    <name>輸出數據庫</name>
    <server>${dbIp}</server>
    <type>REDSHIFT</type>
    <access>Native</access>
    <database>${dbName}</database>
    <port>${dbPort}</port>
    <username>${dbUsername}</username>
    <password>${dbPassword}</password>
    <servername/>
    <data_tablespace/>
    <index_tablespace/>
    <attributes>
      <attribute>
        <code>EXTRA_OPTION_REDSHIFT.tcpKeepAlive</code>
        <attribute>true</attribute>
      </attribute>
      <attribute>
        <code>FORCE_IDENTIFIERS_TO_LOWERCASE</code>
        <attribute>N</attribute>
      </attribute>
      <attribute>
        <code>FORCE_IDENTIFIERS_TO_UPPERCASE</code>
        <attribute>N</attribute>
      </attribute>
      <attribute>
        <code>IS_CLUSTERED</code>
        <attribute>N</attribute>
      </attribute>
      <attribute>
        <code>PORT_NUMBER</code>
        <attribute>5439</attribute>
      </attribute>
      <attribute>
        <code>PRESERVE_RESERVED_WORD_CASE</code>
        <attribute>Y</attribute>
      </attribute>
      <attribute>
        <code>QUOTE_ALL_FIELDS</code>
        <attribute>N</attribute>
      </attribute>
      <attribute>
        <code>SQL_CONNECT</code>
        <attribute>set names utf8;</attribute>
      </attribute>
      <attribute>
        <code>SUPPORTS_BOOLEAN_DATA_TYPE</code>
        <attribute>Y</attribute>
      </attribute>
      <attribute>
        <code>SUPPORTS_TIMESTAMP_DATA_TYPE</code>
        <attribute>Y</attribute>
      </attribute>
      <attribute>
        <code>USE_POOLING</code>
        <attribute>N</attribute>
      </attribute>
    </attributes>
  </connection>
  <order>
  </order>
  <step>
    <name>執行SQL腳本</name>
    <type>ExecSQL</type>
    <description/>
    <distribute>Y</distribute>
    <custom_distribution/>
    <copies>1</copies>
    <partitioning>
      <method>none</method>
      <schema_name/>
    </partitioning>
    <connection>輸出數據庫</connection>
    <execute_each_row>N</execute_each_row>
    <single_statement>N</single_statement>
    <replace_variables>N</replace_variables>
    <quoteString>N</quoteString>
    <sql>INSERT INTO `t_user` ( `name`, `age`) VALUES ('王', '99');</sql>
    <set_params>N</set_params>
    <insert_field/>
    <update_field/>
    <delete_field/>
    <read_field/>
    <arguments>
    </arguments>
    <attributes/>
    <cluster_schema/>
    <remotesteps>
      <input>
      </input>
      <output>
      </output>
    </remotesteps>
    <GUI>
      <xloc>64</xloc>
      <yloc>32</yloc>
      <draw>Y</draw>
    </GUI>
  </step>
  <step_error_handling>
  </step_error_handling>
  <slave-step-copy-partition-distribution>
  </slave-step-copy-partition-distribution>
  <slave_transformation>N</slave_transformation>
  <attributes/>
</transformation>

quartz + kettle8二次開發-實現集羣高可用

文章目錄

背景說明

項目結構

開發步驟

1、引入quartz依賴，和kettle依賴

2、配置quartz

3、編寫系統初始化類，在程序啓動時，添加quartz定時任務

4、編寫quartz任務執行類，這個類中會去調用kettle的方法，從而執行kettle腳本

5、其他類

效果演示

推薦2款開源、美觀的WinForm UI控件庫

NET9 AspnetCore將整合OpenAPI的文檔生成功能而無需三方庫

Java集成kettle開發-解決kettle無法插入空字符串的問題

quartz + kettle8二次開發-實現集羣高可用

Springboot + Drools入門

封裝的influxDB工具類

使用websocket讓服務器端給客戶端推數據

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結