flume-ng 啓動源碼解析

參考文章 https://blog.csdn.net/u011812294/article/details/60878890

bash bin/flume-ng agent --conf-file conf/test-hbase.conf --name agent -Dflume.root.logger=INFO,console &

這個就是我們啓動的命令

這裏最後啓動的是 org.apache.flume.node.Application

Application main

List<LifecycleAware> components = Lists.newArrayList();

        if (reload) { //一般是true
          EventBus eventBus = new EventBus(agentName + "-event-bus");
          PollingPropertiesFileConfigurationProvider configurationProvider =
              new PollingPropertiesFileConfigurationProvider(
                  agentName, configurationFile, eventBus, 30);
          components.add(configurationProvider);
          application = new Application(components);
          eventBus.register(application);
        } else {
          PropertiesFileConfigurationProvider configurationProvider =
              new PropertiesFileConfigurationProvider(agentName, configurationFile);
          application = new Application();
          application.handleConfigurationEvent(configurationProvider.getConfiguration());
        }
      }
      application.start();
備註
不加--no-reload-conf，flume會每隔30秒去重新加載Flume agent的配置文件
如果擔心Flume自動去加載配置文件有時會出現問題，可以在啓動Flume的時候通過加上--no-reload-conf配置來禁止Flume自動加載配置文件只針對apache有效。

參考 https://blog.csdn.net/liuxiao723846/article/details/64128382

挑出主要的代碼說，在這之前還把--conf-file conf/test-hbase.conf --name agent -Dflume.root.logger=INFO,console 這些參數封裝

PollingPropertiesFileConfigurationProvider這個類通過componets.add()放入到了componets裏

然後又把commnents給了application，此時application裏只有components，也就是PollingPropertiesFileConfigurationProvider

PollingPropertiesFileConfigurationProvider

public class PollingPropertiesFileConfigurationProvider
    extends PropertiesFileConfigurationProvider
    implements LifecycleAware {

實現了LifecycleAware接口，注意

Application start()

public void start() {
    lifecycleLock.lock();
    try {
      for (LifecycleAware component : components) {
        supervisor.supervise(component,
            new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);
      }
    } finally {
      lifecycleLock.unlock();
    }
  }

LifecycleSupervisor supervise()

 MonitorRunnable monitorRunnable = new MonitorRunnable();
    monitorRunnable.lifecycleAware = lifecycleAware; //注意這個
    monitorRunnable.supervisoree = process;
    monitorRunnable.monitorService = monitorService;

    supervisedProcesses.put(lifecycleAware, process);

    ScheduledFuture<?> future = monitorService.scheduleWithFixedDelay(
        monitorRunnable, 0, 3, TimeUnit.SECONDS);
    monitorFutures.put(lifecycleAware, future);

new 了一個monitorRunnable，並把start()中的LifecycleAware component的component傳進去

此時這個lifecycleAware就是PollingPropertiesFileConfigurationProvider

MonitorRunnable run()

  public static class MonitorRunnable implements Runnable {

     public void run() {

            switch (supervisoree.status.desiredState) {
              case START:
                try {
                  lifecycleAware.start();
                }
}

 lifecycleAware的實現類就剛好有我上面寫的PollingPropertiesFileConfigurationProvider

PollingPropertiesFileConfigurationProvider start()

 public void start() {
    LOGGER.info("Configuration provider starting");

    Preconditions.checkState(file != null,
        "The parameter file must not be null");

    executorService = Executors.newSingleThreadScheduledExecutor(
            new ThreadFactoryBuilder().setNameFormat("conf-file-poller-%d")
                .build());

    FileWatcherRunnable fileWatcherRunnable =
        new FileWatcherRunnable(file, counterGroup);

    executorService.scheduleWithFixedDelay(fileWatcherRunnable, 0, interval,
        TimeUnit.SECONDS);

    lifecycleState = LifecycleState.START;

    LOGGER.debug("Configuration provider started");
  }

開啓一個延遲30s的調度線程，執行fileWatcherRunnable

FileWatcherRunnable run()

    public void run() {
      LOGGER.debug("Checking file:{} for changes", file);

      counterGroup.incrementAndGet("file.checks");

      long lastModified = file.lastModified();

      if (lastModified > lastChange) {
        LOGGER.info("Reloading configuration file:{}", file);

        counterGroup.incrementAndGet("file.loads");

        lastChange = lastModified;

        try {
          eventBus.post(getConfiguration());
        } catch (Exception e) {
          LOGGER.error("Failed to load configuration data. Exception follows.",
              e);
        } catch (NoClassDefFoundError e) {
          LOGGER.error("Failed to start agent because dependencies were not " +
              "found in classpath. Error follows.", e);
        } catch (Throwable t) {
          // caught because the caller does not handle or log Throwables
          LOGGER.error("Unhandled error", t);
        }
      }
    }

只看try裏的代碼eventBus.post(getConfiguration());

出現了eventbus，這裏解釋下爲啥要去這個名，包括spark裏也有bus這個名稱，說的就是flume想汽車一樣，一批一批的去運載event運到目的地然後放下，再去運下一批

這個eventbus還在哪裏出現了？ application的main方法裏  eventBus.register(application);

getConfiguration()

這個方法是AbstractConfigurationProvider的，這個是PollingPropertiesFileConfigurationProvider 的父類

public MaterializedConfiguration getConfiguration() {

        conf.addChannel(channelName, channelComponent.channel);
        for (Map.Entry<String, SourceRunner> entry : sourceRunnerMap.entrySet()) {
          conf.addSourceRunner(entry.getKey(), entry.getValue());
        }
        for (Map.Entry<String, SinkRunner> entry : sinkRunnerMap.entrySet()) {
          conf.addSinkRunner(entry.getKey(), entry.getValue());
        }
    return conf;
}

我把主要代碼貼出來，就是對conf=>MaterializedConfiguration這個類的屬性 channel sourceRunner sinkrunner 進行封裝了，

封裝的內容就是我們linux上的配好的哪些source sink channel 對象

看到這裏好像還是沒有看到channel sink source 啓動啊 怎麼回事呢？

注意application main 方法裏有

eventBus.register(application);

我們跟蹤到最後的結束方法

   eventBus.post(getConfiguration());

建議看下這邊文章講解eventBus https://www.jianshu.com/p/348ff06f42f6

@Subscribe
  public void handleConfigurationEvent(MaterializedConfiguration conf) {
   try {
      lifecycleLock.lockInterruptibly();
      stopAllComponents();
      startAllComponents(conf); //正式啓動
    }
}

簡單的來說類似消息訂閱模式

eventbus 訂閱了application這個類

eventbus post方法的的屬性是MaterializedConfiguration

@Subscribe 代表這個方法接收eventbus傳遞的 並且是post(conf)的

  例如eventbus.post("string")就只對應 @Subscribe 下 方法入參爲 (String a)的因爲eventbus裏有個typehandler

所以在eventbus.post後 handleConfigurationEvent這個方法被調用了。

startAllComponents(conf)

 private void startAllComponents(MaterializedConfiguration materializedConfiguration) {
    logger.info("Starting new configuration:{}", materializedConfiguration);

    this.materializedConfiguration = materializedConfiguration;

    for (Entry<String, Channel> entry :
        materializedConfiguration.getChannels().entrySet()) {
      try {
        logger.info("Starting Channel " + entry.getKey());
        supervisor.supervise(entry.getValue(),
            new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);
      } catch (Exception e) {
        logger.error("Error while starting {}", entry.getValue(), e);
      }
    }

    /*
     * Wait for all channels to start.
     */
    for (Channel ch : materializedConfiguration.getChannels().values()) {
      while (ch.getLifecycleState() != LifecycleState.START
          && !supervisor.isComponentInErrorState(ch)) {
        try {
          logger.info("Waiting for channel: " + ch.getName() +
              " to start. Sleeping for 500 ms");
          Thread.sleep(500);
        } catch (InterruptedException e) {
          logger.error("Interrupted while waiting for channel to start.", e);
          Throwables.propagate(e);
        }
      }
    }

    for (Entry<String, SinkRunner> entry : materializedConfiguration.getSinkRunners().entrySet()) {
      try {
        logger.info("Starting Sink " + entry.getKey());
        supervisor.supervise(entry.getValue(),
            new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);
      } catch (Exception e) {
        logger.error("Error while starting {}", entry.getValue(), e);
      }
    }

    for (Entry<String, SourceRunner> entry :
         materializedConfiguration.getSourceRunners().entrySet()) {
      try {
        logger.info("Starting Source " + entry.getKey());
        supervisor.supervise(entry.getValue(),
            new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);
      } catch (Exception e) {
        logger.error("Error while starting {}", entry.getValue(), e);
      }
    }

    this.loadMonitoring();
  }

這裏是我們從我們繞了一圈的最後封裝好了的conf裏獲取channel sink source 的對象，然後啓動這些對象的start方法

supervisor.supervise(entry.getValue(),
            new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);