【任務調度】quartz 任務調度框架工作原理源碼分析

Quartz: Java事實標準的任務調度框架，支持本地任務和分佈式任務調度

基於Quartz開源分佈式任務調度框架：XXL-JOB

1. Quartz 基礎概念

Job(作業)：封裝任務執行的業務邏輯
JobDetail：Quartz 對調度任務的抽象，通過JobKey唯一標識
Trigger(觸發器): 封裝調度任務的調度方式，例如：間隔2s觸發等，通過TriggerKey唯一標識，支持CronTrigger、SimpleTrigger
SchedulerFactory(調度器工廠)：調度器工廠類
Scheduler(調度器): 進行任務調度(Trigger + Job)
JobRunShellFactory: JobRunShell工廠類
JobRunShell: 執行調度任務的具體類
QuartzSchedulerThread： Quartz調度線程

觸發器狀態：TriggerState

NONE：觸發器無效
NORMAL: 正常
PAUSED：作業暫停
COMPLETE：完成
ERROR：錯誤
BLOCKED：阻塞

2. Quartz 基礎配置

Quartz 配置源加載順序：

系統屬性：org.quartz.properties(支持配置絕對路徑和classpath路徑配置)
項目classpath根目錄：quartz.properties

Quartz常用配置

org.quartz.jobStore.class -> org.quartz.simpl.RAMJobStore (默認內存保存作業數據，支持數據庫存儲)
org.quartz.scheduler.instanceName -> DefaultQuartzScheduler： 默認調度器實例名稱
org.quartz.scheduler.threadName： 調度線程名，默認<schedName>_QuartzSchedulerThread,默認DefaultQuartzScheduler_QuartzSchedulerThread
org.quartz.scheduler.instanceId：默認NON_CLUSTERED，還支持：AUTO\SYS_PROP
org.quartz.scheduler.instanceIdGenerator： 若實例ID:AUTO, 調度器實例ID生成器，默認SimpleInstanceIdGenerator
org.quartz.scheduler.classLoadHelper.class: 類加載器輔助類，默認：org.quartz.simpl.CascadingClassLoadHelper
org.quartz.scheduler.wrapJobExecutionInUserTransaction：事務內執行作業
org.quartz.scheduler.jobFactory.class： 作業工廠，默認null
org.quartz.scheduler.idleWaitTime： 空閒等待時間
org.quartz.scheduler.dbFailureRetryInterval： 失敗重試時間
org.quartz.scheduler.makeSchedulerThreadDaemon： 調度器線程是否是守護線程
org.quartz.scheduler.rmi.registryHost： quartz RMI主機(提供給外部訪問的主機名)，默認：localhost
org.quartz.scheduler.rmi.registryPort： RMI 端口，默認：1099
org.quartz.scheduler.rmi.createRegistry： RMI創建工廠：never(默認)、always、as_needed
org.quartz.scheduler.rmi.bindName： rmi綁定名稱
org.quartz.jobStore.class： 作業存儲配置，默認：RAMJobStore，若屬於JobStoreSupport類型，例如數據庫則進行數據庫鎖處理
org.quartz.jobStore.lockHandler.class: 鎖處理器
org.quartz.jobStore.tablePrefix：quartz作業數據存儲數據庫表前綴
org.quartz.jobStore.schedName： 作業關聯調度器名稱
org.quartz.dataSource： 數據源名稱
org.quartz.dataSource.<數據源名稱>.connectionProvider.class： 數據源驅動：PoolingConnectionProvider 池化數據庫連接池處理
org.quartz.plugin： quartz插件
org.quartz.plugin.<插件名>： 插件配置
org.quartz.jobListener： 作業監聽器
org.quartz.triggerListener： 觸發器監聽器
org.quartz.threadExecutor.class： 線程池配置，默認：DefaultThreadExecutor，運行QuartzScheduler線程
org.quartz.threadPool.class： 線程池配置，默認：SimpleThreadPool，QuartzScheduler使用的線程
org.quartz.jobStore.isClustered： 是否開啓集羣模式，JobStore必須是數據庫存儲

3. 任務調度

3.1 獲取調度器

源碼：org.quartz.impl.StdSchedulerFactory.getScheduler()

public Scheduler getScheduler() throws SchedulerException {
   
    // 若配置爲空，則初始化Quartz配置
    if (cfg == null) {
        initialize();
    }

    // 調度器倉庫(單例模式)：底層維護Map緩存，key: 調度器名稱 value: 調度器實例
    SchedulerRepository schedRep = SchedulerRepository.getInstance();

    // 緩存查找調度器，若找到則判斷調度器是否關閉，若未關閉直接返回，若關閉則清除緩存
    Scheduler sched = schedRep.lookup(getSchedulerName());
    if (sched != null) {
        if (sched.isShutdown()) {
            schedRep.remove(getSchedulerName());
        } else {
            return sched;
        }
    }
    
    // 調度器初始化
    sched = instantiate();
    return sched;
}

3.2 調度器初始化

源碼(代碼較多)：org.quartz.impl.StdSchedulerFactory.instantiate()

作用：設置調度器使用的相關資源

JobStore： Job作業數據存儲方式，根據org.quartz.jobStore.class 配置，默認RAMJobStore，支持JobStoreSupport，Job作業數據庫存儲，適用於分佈式任務調度
ThreadPool： QuartzScheduler默認使用線程池，根據org.quartz.threadPool.class配置,默認：SimpleThreadPool
QuartzScheduler： Quartz底層任務調度實現類
DBConnectionManager： Job數據數據存儲，數據庫管理器初始化
instanceIdGeneratorClass：調度器ID自動生成器，默認：null
classLoadHelperClass: 類加載工具類，反射工具類

QuartzSchedulerResources：調度器所需資源抽象類

調度器相關變量說明：

schedName：調度器名稱，根據org.quartz.scheduler.instanceName， Quartz默認DefaultQuartzScheduler，若手動quartz.properties未指定屬性則默認QuartzScheduler
threadName：調度線程名稱（QuartzSchedulerThread），根據org.quartz.scheduler.threadName配置，默認：schedName + _QuartzSchedulerThread
schedInstId：調度器QuartzScheduler實例Id,根據org.quartz.scheduler.instanceId配置，默認NON_CLUSTERED，支持：AUTO/SYS_PROP
dbFailureRetry：業務調度失敗重試時間間隔，根據org.quartz.scheduler.dbFailureRetryInterval配置，默認：15s
batchTimeWindow：時間窗口前批量觸發
maxBatchSize：批量拉取Trigger最大數量
interruptJobsOnShutdown：當關閉作業時，中斷作業線程
interruptJobsOnShutdownWithWait：當關閉作業時，等待中斷作業線程
JMX相關配置(默認關閉：與RMI 2選1，對應調度器：RemoteMBeanScheduler)：jmxExport、jmxProxy、jmxProxyClass
RMI相關配置(默認關閉：與JMX 2選1, 對應調度器：RemoteScheduler)：rmiExport、rmiProxy、rmiHost、rmiPort、rmiServerPort、rmiBindName
managementRESTServiceEnabled： quartz rest服務是否開啓，默認false
managementRESTServiceHostAndPort: quartz rest服務主機和端口，默認：9889
plugins： quartz 插件配置，根據org.quartz.plugin配置
jobListeners： quartz 作業監聽器，根據org.quartz.jobListener配置
triggerListener: 觸發器監聽器，根據org.quartz.triggerListener配置
threadExecutor：線程池，默認DefaultThreadExecutor，QuartzScheduler任務調度使用
jobRunShellFactory: JobRunShell工廠類，jobRunShell用於執行具體的作業邏輯，內部包裝Job接口
QuartzSchedulerResources： quartz 調度器資源類，包含任務調度所需所有資源

如果JobStore數據庫存儲，需對數據庫相關進行校驗操作：JobStoreSupport

lockHandlerClass：數據庫分佈式鎖處理器，可根據 org.quartz.jobStore.lockHandler.class 配置

Semaphore接口：

obtainLock：獲取鎖
releaseLock：釋放鎖
requiresConnection：判斷是否需要連接，例如：分佈式任務調度需獲取數據庫連接用於數據庫操作

lockHandlerClass類型：Semaphore

SimpleSemaphore：無需使用數據庫實現鎖，只支持本地任務調度，數據保存在本地JVM內存，通過synchronize實現，本地通過ThreadLocal確定鎖的持有者：lockOwners
JTANonClusteredSemaphore：非quartz集羣模式JTA事務實現鎖機制，JobStore採用：JobStoreCMT
DBSemaphore: StdRowLockSemaphore(行鎖)、UpdateLockRowSemaphore(行鎖，主要用於適配不支持SELECT FOR UPDATE語法數據庫，例如mssql)

分佈式鎖名稱：qrtz_locks

STATE_ACCESS
TRIGGER_ACCESS

QuartzScheduler底層調度器創建：qs = new QuartzScheduler(rsrcs, idleWaitTime, dbFailureRetry);

public QuartzScheduler(QuartzSchedulerResources resources, long idleWaitTime, @Deprecated long dbRetryInterval)
    throws SchedulerException {
    this.resources = resources;

    //添加作業監聽器
    if (resources.getJobStore() instanceof JobListener) {
        addInternalJobListener((JobListener)resources.getJobStore());
    }
    
    // quartz調度線程
    this.schedThread = new QuartzSchedulerThread(this, resources);
    ThreadExecutor schedThreadExecutor = resources.getThreadExecutor();
    schedThreadExecutor.execute(this.schedThread); // 啓動任務調度線程
    if (idleWaitTime > 0) {
        this.schedThread.setIdleWaitTime(idleWaitTime); //設置線程空閒等待時間
    }

    //添加作業監聽器，主要用於監控正在執行的作業，executingJobs緩存維護
    jobMgr = new ExecutingJobsManager();
    addInternalJobListener(jobMgr);

    // 錯誤日誌監聽器：SchedulerListener，監聽調度生命週期,例如實現類：JobRunShell
    errLogger = new ErrorLogger();
    addInternalSchedulerListener(errLogger);

    // 調度器：QuartzScheduler信號通訊
    signaler = new SchedulerSignalerImpl(this, this.schedThread);
    getLog().info("Quartz Scheduler v." + getVersion() + " created.");
}

Scheduler初始化，創建StdScheduler實例： Scheduler scheduler = instantiate(rsrcs, qs);
JobRunShellFactory初始化：JobRunShellFactory綁定調度器：initialize(scheduler)
QuartzScheduler初始化，開啓RMI/JMX支持，默認都開啓：qs.initialize()
緩存Scheduler: schedRep.bind(scheduler);

3.3 調度器啓動

源碼：org.quartz.core.QuartzScheduler.start

 public void start() throws SchedulerException {
    // 調度器已停止或關閉則直接拋出異常
    if (shuttingDown|| closed) {
        throw new SchedulerException("The Scheduler cannot be restarted after shutdown() has been called.");
    }

    // QTZ-212 : calling new schedulerStarting() method on the listeners
    // right after entering start()： 通知SchedulerListener調度器正在啓動
    notifySchedulerListenersStarting();

    // 第1次啓動initialStart爲空，設置調度器啓動時間
    if (initialStart == null) {
        initialStart = new Date();
        this.resources.getJobStore().schedulerStarted();  // 通知JobStore調度器已經啓動          
        startPlugins(); //啓動插件
    } else {
        resources.getJobStore().schedulerResumed();
    }

    //通知主流程循環下個處理點是否暫停
    schedThread.togglePause(false);
    getLog().info("Scheduler " + resources.getUniqueIdentifier() + " started.");
    
    // 通知SchedulerListener調度器已啓動完畢
    notifySchedulerListenersStarted();
}

3.4 註冊作業到調度器

代碼：

scheduler.scheduleJob(jobDetail, trigger);
org.quartz.core.QuartzScheduler.scheduleJob

解釋：將作業和觸發器綁定註冊到調度器

public Date scheduleJob(JobDetail jobDetail, Trigger trigger) throws SchedulerException {
    
    // 校驗調度器是否停止狀態
    validateState();

    /** 省略相關校驗代碼 **/
    Calendar cal = null;
    if (trigger.getCalendarName() != null) { // 日曆類型觸發器
        cal = resources.getJobStore().retrieveCalendar(trigger.getCalendarName());
    }

    // 計算作業第一次觸發時間，實際不會運行，只要用於標識作業開始調度的起始時間,getNextFireTime獲取下一次調度時間
    Date ft = trig.computeFirstFireTime(cal);
    if (ft == null) {
        throw new SchedulerException("Based on configured schedule, the given trigger '" + trigger.getKey() + "' will never fire.");
    }

    // 保存作業數據
    resources.getJobStore().storeJobAndTrigger(jobDetail, trig);

    // 通知SchedulerListener新增作業註冊事件
    notifySchedulerListenersJobAdded(jobDetail);
    
    // 通知下一次調度時間：org.quartz.core.QuartzSchedulerThread.signalSchedulingChange
    notifySchedulerThread(trigger.getNextFireTime().getTime());

    //通知作業已經調度
    notifySchedulerListenersSchduled(trigger);
    return ft;
}

4. quartz 作業調度線程 QuartzSchedulerThread

核心代碼：線程 run 方法

// 獲取待觸發的作業觸發器： 判斷條件，下次觸發時間 < now(當前時間) + idleWaitTime(空閒等待時間) + batchTimeWindow(批量拉取觸發器時間窗口)，相當於任務調度前預先加載，類似緩存預熱的意思
// availThreadCount： 調度任務執行可用線程，通過 org.quartz.threadPool.threadCount 配置
// qsRsrcs.getMaxBatchSize()： 一次性最多調度任務數，通過 org.quartz.scheduler.batchTriggerAcquisitionMaxCount 配置
triggers = qsRsrcs.getJobStore().acquireNextTriggers(now + idleWaitTime, Math.min(availThreadCount, qsRsrcs.getMaxBatchSize()), qsRsrcs.getBatchTimeWindow());

// 通知JobStrore指定觸發器列表將要調度，主要用於更新觸發器的狀態、時間校驗等，便於流程控制
List<TriggerFiredResult> res = qsRsrcs.getJobStore().triggersFired(triggers);

//通過 JobRunShell 真實執行具體作業線程
shell = qsRsrcs.getJobRunShellFactory().createJobRunShell(bndle); // 創建JobRunShell
shell.initialize(qs); // 初始化JobRunShell
qsRsrcs.getThreadPool().runInThread(shell) //運行JobRunShell

4.1 獲取待執行的觸發器 acquireNextTriggers

源碼：org.quartz.simpl.RAMJobStore.acquireNextTriggers(舉例：RAmStore)

public List<OperableTrigger> acquireNextTriggers(long noLaterThan, int maxCount, long timeWindow) {
    synchronized (lock) {
        List<OperableTrigger> result = new ArrayList<OperableTrigger>();
        Set<JobKey> acquiredJobKeysForNoConcurrentExec = new HashSet<JobKey>();
        Set<TriggerWrapper> excludedTriggers = new HashSet<TriggerWrapper>();
        long batchEnd = noLaterThan;
        
        // return empty list if store has no triggers.
        if (timeTriggers.size() == 0)
            return result;
        
        while (true) {
            TriggerWrapper tw;

            try {
                // 獲取第1個觸發器，第1次註冊作業時會添加到timeTriggers，timeTriggers維護者可能執行的觸發器
                tw = timeTriggers.first();
                if (tw == null)
                    break;
                timeTriggers.remove(tw); // 從timeTriggers清除，避免重複處理
            } catch (java.util.NoSuchElementException nsee) {
                break;
            }

            // 觸發器下次執行時間爲空，則跳過不處理
            if (tw.trigger.getNextFireTime() == null) {
                continue;
            }

            // 觸發器失效時間校驗
            if (applyMisfire(tw)) {
                if (tw.trigger.getNextFireTime() != null) {
                    timeTriggers.add(tw);
                }
                continue;
            }
            
            // 重點：如哦觸發器下次執行時間大於 batchEnd，則放入timeTriggers等待下一次執行
            // 分析：相當於提前加載觸發器，例如：提前10s加載，若超過10s則等待下一輪處理，類似Nacos配置長輪詢機制，考慮網絡抖動原因，長輪詢默認30s,實際服務端提前500毫秒就返回
            if (tw.getTrigger().getNextFireTime().getTime() > batchEnd) {
                timeTriggers.add(tw);
                break;
            }
            
            // 緩存獲取觸發器綁定的作業Job
            JobKey jobKey = tw.trigger.getJobKey();
            // 獲取作業Job
            JobDetail job = jobsByKey.get(tw.trigger.getJobKey()).jobDetail;

            // 是否允許併發執行，作業上標註：@DisallowConcurrentExecution 註解
            if (job.isConcurrentExectionDisallowed()) {
                if (acquiredJobKeysForNoConcurrentExec.contains(jobKey)) {
                    excludedTriggers.add(tw);
                    continue; // go to next trigger in store.
                } else {
                    acquiredJobKeysForNoConcurrentExec.add(jobKey);
                }
            }
            
            // 更新觸發器狀態：STATE_ACQUIRED
            tw.state = TriggerWrapper.STATE_ACQUIRED;
            tw.trigger.setFireInstanceId(getFiredTriggerRecordId());
            OperableTrigger trig = (OperableTrigger) tw.trigger.clone();
            if (result.isEmpty()) {
                batchEnd = Math.max(tw.trigger.getNextFireTime().getTime(), System.currentTimeMillis()) + timeWindow;
            }
            result.add(trig);
            if (result.size() == maxCount)
                break;
        }

        // If we did excluded triggers to prevent ACQUIRE state due to DisallowConcurrentExecution, we need to add them back to store.
        if (excludedTriggers.size() > 0)
            timeTriggers.addAll(excludedTriggers);

        // 返回下個時間窗口內待觸發的觸發器
        return result;
    }
}

4.2 執行具體作業 JobRunShell

源碼：JobRunShellFactory 默認 JTAAnnotationAwareJobRunShellFactory

shell = qsRsrcs.getJobRunShellFactory().createJobRunShell(bndle); // 創建JobRunShell
shell.initialize(qs); // 本質JobExecutionContextImpl封裝作業： this.jec = new JobExecutionContextImpl(scheduler, firedTriggerBundle, job);
qsRsrcs.getThreadPool().runInThread(shell)： 線程池執行具體的作業邏輯

創建JobRunShell

public JobRunShell createJobRunShell(TriggerFiredBundle bundle) throws SchedulerException {
    
    //判斷作業實現類是否再JTA事務執行： 作業類上註解：@ExecuteInJTATransaction
    ExecuteInJTATransaction jtaAnnotation = ClassUtils.getAnnotation(bundle.getJobDetail().getJobClass(), ExecuteInJTATransaction.class);
    if(jtaAnnotation == null)
        return new JobRunShell(scheduler, bundle);
    else { //JTA事務處理
        int timeout = jtaAnnotation.timeout();
        if (timeout >= 0) {
            return new JTAJobRunShell(scheduler, bundle, timeout);
        } else {
            return new JTAJobRunShell(scheduler, bundle);
        }
    }
}

JobRunShell 初始化

public void initialize(QuartzScheduler sched) throws SchedulerException {
    this.qs = sched;
    Job job = null;
    JobDetail jobDetail = firedTriggerBundle.getJobDetail(); //獲取作業信息

    try {
        job = sched.getJobFactory().newJob(firedTriggerBundle, scheduler); // 獲取真實的作業邏輯：實現Job接口
    } catch (SchedulerException se) {
        sched.notifySchedulerListenersError(
                "An error occured instantiating job to be executed. job= '"
                        + jobDetail.getKey() + "'", se);
        throw se;
    } catch (Throwable ncdfe) { // such as NoClassDefFoundError
        SchedulerException se = new SchedulerException(
                "Problem instantiating class '"
                        + jobDetail.getJobClass().getName() + "' - ", ncdfe);
        sched.notifySchedulerListenersError(
                "An error occured instantiating job to be executed. job= '"
                        + jobDetail.getKey() + "'", se);
        throw se;
    }
    // 將作業執行的相關邏輯封裝到JobExecutionContextImpl
    this.jec = new JobExecutionContextImpl(scheduler, firedTriggerBundle, job);
}

JobRunShell執行：真實業務邏輯執行源碼：org.quartz.core.JobRunShell.run

public void run() {
    // 吧JobRunShell自生加入SchedulerListener，方便監聽作業執行的生命週期
    qs.addInternalSchedulerListener(this);
    try {
        OperableTrigger trigger = (OperableTrigger) jec.getTrigger();
        JobDetail jobDetail = jec.getJobDetail();
        do {

            JobExecutionException jobExEx = null;
            Job job = jec.getJobInstance(); // 獲取正式作業執行類：實現job接口

            try {
                begin();
            } catch (SchedulerException se) {
                qs.notifySchedulerListenersError("Error executing Job ("
                        + jec.getJobDetail().getKey()
                        + ": couldn't begin execution.", se);
                break;
            }

            // notify job & trigger listeners... 通知jobListener和TiggerListener 作業準備開始執行
            try {
                if (!notifyListenersBeginning(jec)) {
                    break;
                }
            } catch(VetoedException ve) {
                try {
                    CompletedExecutionInstruction instCode = trigger.executionComplete(jec, null);
                    qs.notifyJobStoreJobVetoed(trigger, jobDetail, instCode);
                    
                    // QTZ-205
                    // Even if trigger got vetoed, we still needs to check to see if it's the trigger's finalized run or not.
                    if (jec.getTrigger().getNextFireTime() == null) {
                        qs.notifySchedulerListenersFinalized(jec.getTrigger());
                    }

                    complete(true);
                } catch (SchedulerException se) {
                    qs.notifySchedulerListenersError("Error during veto of Job ("
                            + jec.getJobDetail().getKey()
                            + ": couldn't finalize execution.", se);
                }
                break;
            }

            long startTime = System.currentTimeMillis();
            long endTime = startTime;

            // execute the job
            try {
                log.debug("Calling execute on job " + jobDetail.getKey());
                job.execute(jec); // 執行Job實現類邏輯，由使用quartz框架的項目自己手動實現
                endTime = System.currentTimeMillis();
            } catch (JobExecutionException jee) {
                endTime = System.currentTimeMillis();
                jobExEx = jee;
                getLog().info("Job " + jobDetail.getKey() + " threw a JobExecutionException: ", jobExEx);
            } catch (Throwable e) {
                endTime = System.currentTimeMillis();
                getLog().error("Job " + jobDetail.getKey() + " threw an unhandled Exception: ", e);
                SchedulerException se = new SchedulerException("Job threw an unhandled exception.", e);
                qs.notifySchedulerListenersError("Job ("
                        + jec.getJobDetail().getKey()
                        + " threw an exception.", se);
                jobExEx = new JobExecutionException(se, false);
            }

            jec.setJobRunTime(endTime - startTime); // 設置作業執行時間

            // notify all job listeners
            if (!notifyJobListenersComplete(jec, jobExEx)) { // 通知JobListener作業執行完成
                break;
            }

            CompletedExecutionInstruction instCode = CompletedExecutionInstruction.NOOP;
            // update the trigger
            try {
                instCode = trigger.executionComplete(jec, jobExEx);
            } catch (Exception e) {
                // If this happens, there's a bug in the trigger...
                SchedulerException se = new SchedulerException("Trigger threw an unhandled exception.", e);
                qs.notifySchedulerListenersError("Please report this error to the Quartz developers.", se);
            }

            // notify all trigger listeners
            if (!notifyTriggerListenersComplete(jec, instCode)) {
                break;
            }

            // update job/trigger or re-execute job
            if (instCode == CompletedExecutionInstruction.RE_EXECUTE_JOB) {
                jec.incrementRefireCount();
                try {
                    complete(false);
                } catch (SchedulerException se) {
                    qs.notifySchedulerListenersError("Error executing Job ("
                            + jec.getJobDetail().getKey()
                            + ": couldn't finalize execution.", se);
                }
                continue;
            }

            try {
                complete(true);
            } catch (SchedulerException se) {
                qs.notifySchedulerListenersError("Error executing Job ("
                        + jec.getJobDetail().getKey()
                        + ": couldn't finalize execution.", se);
                continue;
            }

            qs.notifyJobStoreJobComplete(trigger, jobDetail, instCode); // 通知JobStore作業執行完成
            break;
        } while (true);

    } finally {
        // 作業處理完成後將JobRunShell自生從SchedulerListener清除
        qs.removeInternalSchedulerListener(this);
    }
}

【任務調度】quartz 任務調度框架工作原理源碼分析

1. Quartz 基礎概念

2. Quartz 基礎配置

3. 任務調度

3.1 獲取調度器

3.2 調度器初始化

3.3 調度器啓動

3.4 註冊作業到調度器

4. quartz 作業調度線程 QuartzSchedulerThread

4.1 獲取待執行的觸發器 acquireNextTriggers

4.2 執行具體作業 JobRunShell

Debian 系統初體驗

vue2數據雙向綁定Object.defineProperty

魔獸爭霸卡頓解決

BCS2024｜Baidu Comate：以研發提效爲驅動實現“安全左移”

Puppeteer實戰案例：自動化抓取社交媒體上的媒體資源

Mac下配置sublime實現LaTeX

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結