Linux Workflow Airflow CeleryExecutor 踩坑心酸歷程

本篇所使用的系統環境爲 Centos 6.8，Python 3.6.5，Airflow 1.9.0

Airflow 的搭建可以參考從0到1，AirFlow搭建踩坑歷程

安裝RabbitMQ

    1.安裝erlang

        erlang地址：http://www.rabbitmq.com/releases/erlang/ 

          使用 rpm -ihv 安裝

    2.安裝RabbitMQ

        rabbitmq地址：http://www.rabbitmq.com/releases/rabbitmq-server/

        我所使用的是 rabbitmq-server-3.3.5-1.noarch.rpm

        yum install -y rabbitmq-server-3.3.5-1.noarch.rpm

    3.配置RabbitMQ

        安裝成功後使用命令 rabbitmqctl status 檢查RabbitMQ 是否正常啓動       

# rabbitmqctl status
Status of node 'rabbit@ip-10-3-0-102' ...
[{pid,76168},
 {running_applications,
     [{rabbitmq_management,"RabbitMQ Management Console","3.3.5"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.3.5"},
      {rabbit,"RabbitMQ","3.3.5"},
      {os_mon,"CPO  CXC 138 46","2.2.7"},
      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.3.5"},
      {webmachine,"webmachine","1.10.3-rmq3.3.5-gite9359c7"},
      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.3.5-git680dba8"},
      {amqp_client,"RabbitMQ AMQP Client","3.3.5"},
      {xmerl,"XML parser","1.2.10"},
      {inets,"INETS  CXC 138 49","5.7.1"},
      {mnesia,"MNESIA  CXC 138 12","4.5"},
      {sasl,"SASL  CXC 138 11","2.1.10"},
      {stdlib,"ERTS  CXC 138 10","1.17.5"},
      {kernel,"ERTS  CXC 138 10","2.14.5"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:48:48] [rq:48] [async-threads:30] [kernel-poll:true]\n"},
 {memory,
     [{total,109340632},
      {connection_procs,5408},
      {queue_procs,49248},
      {plugins,282768},
      {other_proc,9454312},
      {mnesia,70952},
      {mgmt_db,53224},
      {msg_index,47728},
      {other_ets,1168112},
      {binary,6037264},
      {code,17795196},
      {atom,1621321},
      {other_system,72755099}]},
 {alarms,[]},
 {listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,27013577113},
 {disk_free_limit,50000000},
 {disk_free,821105397760},
 {file_descriptors,
     [{total_limit,102542},
      {total_used,5},
      {sockets_limit,92285},
      {sockets_used,1}]},
 {processes,[{limit,1048576},{used,226}]},
 {run_queue,0},
 {uptime,83573}]
...done.

        打開和關閉服務

                關閉服務：

# service rabbitmq-server stop
Stopping rabbitmq-server: rabbitmq-server.

                 打開服務：

service rabbitmq-server start
Starting rabbitmq-server: SUCCESS
rabbitmq-server.

        添加用戶並配置權限

rabbitmqctl add_user admin 123456
rabbitmqctl set_user_tag admin administrator
rabbitmq add_vhost / 
rabbitmqctl set_permissions -p / admin ".*" ".*" ".*" 

        RabbitMQ UI訪問，默認端口是15672：
        http://localhost:15672

        Username：admin

        Password：123456

        可以通過 rabbitmqctl -h 查看相應命令進行操作，也可以在 UI 中進行操作。

 

配置Airflow CeleryExecutor

        pip3 install celery 
        配置CeleryExecutor (rabbitmq支持) 

 

 

        安裝airflow的celery和rabbitmq組件             

 

 

pip3 install airflow[celery,rabbitmq]

     配置 airflow.cfg

# 這裏我將vhost 設置爲 /
broker_url = pyamqp://admin:123456@localhost:5672//
celery_result_backend = db+mysql://airflow:airflow@locahost:3306/airflow
# 將 executor 設置爲 CeleryExecutor
executor = CeleryExecutor
# 設置 default_queue
default_queue = airflow

        重新啓動 airflow

airflow webserver
airflow worker
airflow flower

        Airflow webserver UI 端口號爲 8080

        Airflow flower UI 端口號爲 5555

 

 

    測試 CeleryExecutor

遇到的坑：

    1.queued messages 一直處於ready，沒有消費者進行消費

    執行命令 rabbitmqctl list_consumers 查看是否有消費者，發現消費者不處於正常隊列中。

    或者也可以在頁面中點擊你所創建的queue，判斷是否有consumer。

如果沒有，可以通過手動的方式進行 bind.

參考資料   

        1.Using RabbitMQ

        2.Airflow Scaling Out with Celery