celery初識

celery異步任務

參考博客

• 在Linux下安裝：pip install celery（安裝redis：yum install redis-server）據說celery在Windows不好使

• 小栗子：task1.py

  from celery import Celery
  import time
  
  app = Celery(
              'tasks',   # 任務名
              broker='redis://localhost:6379',   # 任務存儲
              backend='redis://localhost:6379')  # 任務返回存儲
  
  @app.task
  def hello(a,b):
      print('running tasks...')
      time.sleep(10)
      c=a+b
      return 'hello world%d'%c
  

  • 執行任務：celery -A task1 worker -l debug
  • 在Python解釋器中

    >>> import task1
    >>> t=task1.hello.delay(1,2)
    

    1. t.get()獲取任務結果，有異常時直接報錯。
    2. t.get(timeout=1)超時後沒有結果直接報socket.timeout
    3. t.get(propagte=False)任務執行報錯時只返回錯誤對象
    4. t.ready()檢查任務是否執行完畢
    5. t.traceback查看異常信息

• 在項目中使用celery

  • 目錄結構

    celery_proj/__init__.py
        /celery.py
        /task1.py
        /task2.py
    

  • 飯粒1：

  # celery.py
  # 導入__future__用於指定從絕對路徑導入模塊
  from __future__ import absolute_import, unicode_literals
  from celery import Celery
   
  app = Celery('celery_proj',
              broker='redis://localhost:6379',   # 任務存儲
              backend='redis://localhost:6379',
              include=['celery_proj.task1','celery_proj.task2'])  # 包含多個任務
   
  # Optional configuration, see the application user guide.
  app.conf.update(
  	# 結果保存時間
      result_expires=3600,
  )
   
  if __name__ == '__main__':
      app.start()
  

  # task1.py
  from __future__ import absolute_import, unicode_literals
  from .celery import app
  
  
  @app.task
  def add(x, y):
      return x + y
  
  
  @app.task
  def mul(x, y):
      return x * y
  
  
  @app.task
  def xsum(numbers):
      return sum(numbers)
  

  # task2.py
  from __future__ import absolute_import, unicode_literals
  from .celery import app
  import time, random
  
  @app.task
  def rannum(x, y):
      time.sleep(6)
      return random.randint(x, y)
  

  • 啓動worker：celery -A proj worker -l info
  • 後臺啓動worker：celery multi start/stop/restart [任務名] -A proj -l info --pidfile=proj.pid --logfile=proj.log

• Celery 定時任務

  • 繼續用上慄的目錄結構，目錄內新增periodic_task.py

  from __future__ import absolute_import, unicode_literals
  from .celery import app
  from celery.schedules import crontab
  
  
  @app.on_after_configure.connect
  def setup_periodic_tasks(sender, **kwargs):
      # Calls test('hello') every 10 seconds.
      sender.add_periodic_task(10.0, test.s('hello'), name='add every 10')
  
      # Calls test('world') every 30 seconds
      sender.add_periodic_task(30.0, test.s('world'), expires=10)
  
      # Executes every Monday morning at 7:30 a.m.
      sender.add_periodic_task(
          crontab(hour=14, minute=39),
          test.s('Happy Mondays!'),
      )
  
  
  @app.task
  def test(arg):
      print(arg)
  

  • celery.py中include加入’celery_proj.periodic_task’，加入以下內容：(每5秒執行一次task1.add)

  app.conf.beat_schedule = {
      'add-every-5-seconds': {
          'task': 'celery_proj.task1.add',
          'schedule': 5.0,
          'args': (16, 16)
      },
  }
  app.conf.timezone = 'UTC'
  

  • 啓動任務celery -A celery_proj worker -l debug
  • 啓動定時任務celery -A celery_proj.periodic_task beat -l debug
  • 注意：如果啓動定時任務輸出beat: Waking up in 5.00 minutes.則表示定時任務啓動失敗，原因有可能是配置有問題，也有可能環境衝突。檢查配置無誤後可刪除當前目錄下celerybeat-schedule文件和redis中所有key -> 進入redis-cli執行flushall再重試。

• 在Django中使用celery

  1. 在desk_demo/settings.py中添加celery相關配置

     # celery配置
     CELERY_BROKER_URL = 'redis://localhost:6379'
     CELERY_RESULT_BACKEND = 'redis://localhost:6379'
     

  2. 在settings.py的同級目錄中創建celery.py文件：

     from __future__ import absolute_import, unicode_literals
     import os
     from celery import Celery
     
     # set the default Django settings module for the 'celery' program.
     os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'desk_demo.settings')
     
     # celery名稱可自定義
     app = Celery('proj')
     
     # 將settings.py中前綴是`CELERY_`看做是celery配置信息
     app.config_from_object('django.conf:settings', namespace='CELERY')
     
     # 收集所有app下的任務
     app.autodiscover_tasks()
     
     
     @app.task(bind=True)
     def debug_task(self):
         print('Request: {0!r}'.format(self.request))
     

  3. 在app下創建tasks.py -> 注意必須且只能以此命名

     from __future__ import absolute_import, unicode_literals
     from celery import shared_task
     # shared_task表示其他app也可以使用此任務
     
     @shared_task
     def add(x, y):
         return x + y
     
     
     @shared_task
     def mul(x, y):
         return x * y
     
     
     @shared_task
     def xsum(numbers):
         return sum(numbers)
     

  4. 配置全局urls.py

     from app.app01.views import *
     urlpatterns = [
     	...
         path('celery_call/', celery_call),
         path('celery_result/', celery_result),
     	...]
     

  5. 配置views.py

     import random
     # from extra_app.helpdesk import tasks
     from app.app01 import tasks
     from django.shortcuts import render, HttpResponse
     from celery.result import AsyncResult
     
     
     def celery_call(request):
     	# 獲取task_id視圖
         num = random.randint(1, 10000)
         t = tasks.randnum.delay(1, num)
         return HttpResponse(t.id)
     
     
     def celery_result(request):
     	# 獲取結果視圖
         task_id = request.GET.get('id')
         res = AsyncResult(id=task_id)
         if res.ready():
             return HttpResponse(res.get())
         else:
             return HttpResponse(res.ready())
     

• 使用django-celery-beat配置定時任務

  1. 安裝：pip install django-celery-beat
  2. 寫入settings.py中INSTALLED_APPS：'django_celery_beat',
  3. 生成數據庫結構：python manage.py migrate
  4. 在admin界面配置定時任務
  5. 啓動worker：celery -A desk_demo worker -l info
  6. 執行定時任務：celery -A desk_demo beat -l info -S django
  • 注意：在admin界面新增任務Periodic tasks時Schedule中的Interval、Crontab、Solar三選一