Django筆記 使用Celery來進行定時Batch任務

環境

unix
Python2.7
Django1.8
Celery3.1
django-celery3.1

需要安裝

pip install celery
pip install django-celery

代碼

代碼的框架

djangotest
  └-apptest
      └-tasks.py
      └-

  └-djangotest
      └-celery.py
      └-settings.py

代碼

__init__.py
from __future__ import absolute_import

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app  # noqa

djangotest/setting.py

# 30秒進行一次處理
from datetime import timedelta
CELERYBEAT_SCHEDULE = {
    'add-every-30-seconds': {
        'task': 'apptest.tasks.test_celery',
        'schedule': timedelta(seconds=30),
        'args': (16, 16)
    },
    # Executes every Monday morning at 7:30 A.M
    'add-every-1-minute': {
        'task': 'apptest.tasks.test_celery3',
        'schedule': crontab(minute='*/1'),
        'args': ('test_celery3',),
    },
}

CELERY_TIMEZONE = 'UTC'

BROKER_URL = 'django://'

djangotest/celery.py

from __future__ import absolute_import

import os

from celery import Celery

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'djangotest.settings')

from django.conf import settings

app = Celery('djangotest')

# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

app.conf.update(
    CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend',
)

app.conf.update(
    CELERY_RESULT_BACKEND='djcelery.backends.cache:CacheBackend',
)

@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

apptest/tasks.py

from __future__ import absolute_import

import logging
from celery import task
from celery.utils.log import get_task_logger
from appmanager import models

@task
def test_celery(x, y):
    logger = get_task_logger(__name__)
    logger.info('func start  ----------------->')
    logger.info('application:%s', "TEST_APP")
    logger.info('func end -------------------->')
    return x + y

在setting.py中添加app和同步數據庫

INSTALLED_APPS = [
    。。。
    'djcelery',
    'kombu.transport.django',
    'apptask',
]

執行
python manage.py migrate

啓動celery服務

1.啓動工作進程 work

celery -A djangotest worker -B -l info

#error ImportError: No module named 
celery -A djangotest worker --app=djangotest.celery:app -l info -c 1

python manage.py celeryd -B 只啓動一個進程

2.啓動工作觸發進程 celerybeat

python manage.py celerybeat

然後就會循環顯示執行的結果

func start  ----------------->
application:TEST_APP
func end -------------------->

在生產環境中，使用django作爲容器，並不安定

1.安裝redis和啓動
2.在setting文件中，設置容器地址。

BROKER_URL = 'redis://localhost:6379/0'

3.在celery setting文件中，設

CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'

參照

http://www.weiguda.com/blog/73/
http://www.tuicool.com/articles/aIvYbie
http://qiita.com/shinno21/items/8c57a66536c0f38b3fbb

http://michal.karzynski.pl/blog/2014/05/18/setting-up-an-asynchronous-task-queue-for-django-using-celery-redis/

http://www.ibm.com/developerworks/cn/opensource/os-cn-celery-web-service/index.html?ca=drs-&utm_source=tuicool&utm_medium=referral

djcelery入門：實現運行定時任務 http://my.oschina.net/kinegratii/blog/292395#OSC_h2_6

使用celery之瞭解celery - 小明明’ http://www.dongwm.com/archives/shi-yong-celeryzhi-liao-jie-celery/?utm_source=tuicool&utm_medium=referral

使用django+celery+RabbitMQ實現異步執行 http://charlee.li/django-celery-rabbitmq-intro.html

官方文檔：http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html

cron時間格式：http://www.nncron.ru/help/EN/working/cron-format.htm

celery最佳實踐：http://my.oschina.net/siddontang/blog/284107

使用django+celery+RabbitMQ實現異步執行：http://blog.charlee.li/django-celery-rabbitmq-intro/

RabbitMQ文檔： http://www.rabbitmq.com/documentation.html

Django/Celery Quickstart (or, how I learned to stop using cron and love celery) http://chase-seibert.github.io/blog/2010/07/09/djangocelery-quickstart-or-how-i-learned-to-stop-using-cron-and-love-celery.html

Using Celery to handle asynchronous tasks in Django – Sebastian Dahlgren http://sebastiandahlgren.se/2012/11/13/using-celery-for-asynchronous-messages-in-django/

Django-celery配置及使用指南 http://zhujinliang.cn/django/2013/07/18/Django-celery%E9%85%8D%E7%BD%AE%E5%8F%8A%E4%BD%BF%E7%94%A8%E6%8C%87%E5%8D%97/

Introducing Celery for Python+Django - LINUX For You http://www.opensourceforu.com/2013/12/introducing-celery-pythondjango/

Django中如何使用django-celery完成異步任務 (1) | 上海味股達信息科技有限公司 http://www.weiguda.com/blog/73/
Django中如何使用django-celery完成異步任務 (2) http://www.weiguda.com/blog/74/

Django Celery Architecture | langoor.mobi Blog http://blog.langoor.mobi/django-celery-redis-vs-rabbitmq-message-broker/django_celery_architecture/

Queueing Messages using Celery with RabbitMQ Message Broker Server - 2014 http://www.bogotobogo.com/python/RabbitMQ_Celery/python_Queueing%20using_Celery_with_RabbitMQ_Message_Broker_Server.php

AMQP, RabbitMQ and Celery - A Visual Guide For Dummies | Abhishek Tiwari http://abhishek-tiwari.com/post/amqp-rabbitmq-and-celery-a-visual-guide-for-dummies

Tracing Celery Performance For Web Applications - 推酷 http://www.tuicool.com/articles/ZbyUnu

http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html

redis

在Ubuntu中安裝Redis http://blog.fens.me/linux-redis-install/

http://dim5.net/windows-server/install-redis.html
http://www.cnblogs.com/happyframework/archive/2013/07/18/3197392.html