docker下載、啓動、配置apache airflow

        mac下安裝apache airflow，會遇到各種麻煩，如python版本不支持、網絡下載包慢、各種配置等繁瑣問題，幸好docker上有airflow資源。廢話不多說，接下來記錄我在本地搭建airflow項目的過程。

 

一、啓動本地docker

二、訪問docker hub上的airflow官網項目地址：https://hub.docker.com/r/apache/airflow。

$docker pull apache/airflow

三、在本地命令行工具執行docker pull apache/airflow，執行完之後，等待資源下載完成，之後重啓docker容器，即可看到docker已經有airflow鏡像。

四、運行airflow，在命令行工具執行以下命令（也可以在docker上對應的鏡像鏡像圖像操作）       

$docker run -d -p 8080:8080 puckel/docker-airflow

五、如第四步沒什麼問題，即可訪問到airflow控制檯。

       訪問地址：http://localhost:8080/admin/。（端口號在啓動命令行裏設置）

        注：還沒初始化數據庫之前，dag下的任務列表是爲空的。

六、初始化數據庫        

$airflow initdb

這時候應該會報以下錯誤：

$ airflow initdb
DB: sqlite:////usr/local/airflow/airflow.db
[2020-05-21 12:09:20,775] {{db.py:368}} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/crypto.py", line 91, in get_fernet
    for fernet_part in fernet_key.split(',')
  File "/usr/local/lib/python3.7/site-packages/airflow/models/crypto.py", line 91, in <listcomp>
    for fernet_part in fernet_key.split(',')
  File "/usr/local/lib/python3.7/site-packages/cryptography/fernet.py", line 35, in __init__
    key = base64.urlsafe_b64decode(key)
  File "/usr/local/lib/python3.7/base64.py", line 133, in urlsafe_b64decode
    return b64decode(s)
  File "/usr/local/lib/python3.7/base64.py", line 87, in b64decode
    return binascii.a2b_base64(s)
binascii.Error: Incorrect padding

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 37, in <module>
    args.func(args)
  File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 1140, in initdb
    db.initdb(settings.RBAC)
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 117, in initdb
    schema='default'))
  File "<string>", line 4, in __init__
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 441, in _initialize_instance
    manager.dispatch.init_failure(self, args, kwargs)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 438, in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/connection.py", line 127, in __init__
    self.extra = extra
  File "<string>", line 1, in __set__
  File "/usr/local/lib/python3.7/site-packages/airflow/models/connection.py", line 218, in set_extra
    fernet = get_fernet()
  File "/usr/local/lib/python3.7/site-packages/airflow/models/crypto.py", line 95, in get_fernet
    raise AirflowException("Could not create Fernet object: {}".format(ve))
airflow.exceptions.AirflowException: Could not create Fernet object: Incorrect padding

airflow.exceptions.AirflowException: Could not create Fernet object: Incorrect padding

這問題是因爲fernet key不正確，解決方法就是在docker命令行下生成fernet key，並配置到airflow.cfg的fernet_key配置項：

$python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"（回車）

>>>print(Fernet.generate_key().decode())"（回車）
>>>這裏會得到一個字符串，並拷貝這個字符串

---------------------------------------------------

以上配置完成之後，重新執行：

$airflow initdb

如出現以下信息，則表明數據庫初始化成功：

參考：https://www.jianshu.com/p/5349168dd346

 

七、重新刷新airflow控制檯之後：

完！