How to schedule ‘the Boring Stuff’ with Django and Celery Beat

Work of software developers is filled with generating periodic reports, handling vasty imports or exports, backups, frequent API requests, or simply flicking batches of emails. Wouldn’t it be a developer’s paradise to have all these tasks automated and perfectly scheduled? It would — luckily, with help comes Celery Beat. It combines , a well-known task delegation tool, with a nifty scheduler called . In this guide, you will find out . Let’s get to work! What is Celery Beat? Celery Beat how it can help you manage even the most tedious of tasks How to start working with Celery? First of all, . Its latest version (4.2) still supports , but since the new ones won’t, you’ll need to have Celery running Python 2.7 it’s recommended to use Python 3 if you want to work with Celery. For the purpose of this article, . Be aware that Celery is a project with minimal funding and doesn’t support Microsoft Windows. I’m running Django 2.0.6 from Python 3.6.5 image in Docker container To get Celery on wheels, you’ll need a broker to send and receive messages. My choice is , which also is deployed in a container. An alternative for Redis is — you can read more about it in . Redis 4.0.10 RabbitMQ the official Celery documentation Redis has a reputation of being , what makes my choice simple. With Redis installed you just call ‘redis-server’ to run it. That’s it. Here’s a quick test: far easier to install and administrate $ redis-cli pingPONG In the next step, you need to ensure that . PIP is handy to get them in place. At the later stage, you’ll also use benefits of . either your virtual environment or container are equipped with packages: celery 2.10.6 4.20 and redis django_celery_beat==1.1.1 How to use Celery Beat? Celery config may be tricky at times even for top software developers. In this guide I’ll work with : the standard Django structure proj /manage.py /proj/__init__.py /apps.py /celery.py /tasks.py /settings.py /my_app/__init__.py /apps.py /tasks.py Remember to have all . Additionally, each app __init__.py should point to default config: apps.py files setup default_app_config = 'my_app.apps.MyAppConfig' An exception is __init__.py in your project folder, which stores : Celery configuration from __future__ import absolute_import, unicode_literalsfrom .celery import app as celery_app __all__ = ('celery_app',) At this point in the configuration, the next step is to move on to : celery.py from __future__ import absolute_import, unicode_literals import os from celery import Celery # Set default Django settings os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings') app = Celery('proj') # Celery will apply all configuration keys with defined namespace app.config_from_object('django.conf:settings', namespace='CELERY') # Load tasks from all registered apps app.autodiscover_tasks() The other way of configuration, besides using ‘config_from_object’, is — for further details. assigning config values directly within the app check out the documentation With basics taken care of, let’s move on to settings.py and . The most simple config will look like this: set up the Celery Beat’s specifics CELERY_BROKER_URL = 'redis://localhost:6379' # If time zones are active (USE_TZ = True) define your local CELERY_TIMEZONE = 'Europe/Warsaw' # We're going to have our tasks rolling soon, so that will be handy CELERY_BEAT_SCHEDULE = {} There is new lowercase settings standard introduced in the latest versions of Celery Beat. However, there’s no rush in updating it as . this well-adapted format is still supported Scheduling tasks in Celery Beat Well done so far — now it’s time to move to . tasks.py in your app module from celery import task from celery import shared_task # We can have either registered task @task(name='summary') def send_import_summary(): # Magic happens here ... # or @shared_task def send_notifiction(): print(‘Here I\’m’) # Another trick If the tasks you write will serve in reusable apps that can’t depend on the project itself, then app instance can’t be imported directly. In this case, using is the right way to ensure you’ll have everything in place. No matter if a task is registered or shared, Celery is able to discover it. @shared_task decorator With tasks’ functions ready, : let’s move back to settings.py to add a schedule from celery.schedules import crontab CELERY_BROKER_URL = 'redis://localhost:6379' CELERY_TIMEZONE = 'Europe/Warsaw' # Let's make things happen CELERY_BEAT_SCHEDULE = { 'send-summary-every-hour': { 'task': 'summary', # There are 4 ways we can handle time, read further 'schedule': 3600.0, # If you're using any arguments 'args': (‘We don’t need any’,), }, # Executes every Friday at 4pm 'send-notification-on-friday-afternoon': { 'task': 'my_app.tasks.send_notification', 'schedule': crontab(hour=16, day_of_week=5), }, } In the example above, you’ve used two methods of handling time: in seconds to declare an interval between repetitions; simple time representation , which works as ‘run every’ and lets you specify an exact time, day of week/month, and month in the year. crontab Also well know can be applied here to set ‘by the clock’ schedule. The other fancy option which Celery Beat brings up is : Python timedelta method task execution according to the sun activity from celery.schedules import solar # [...] 'schedule': solar('sunset', -37.81753, 144.96715), Time to run your first worker! Settings are done and dusted. Let’s give them a try. $ celery -A proj beat -l INFO # For deeper logs use DEBUG Beat can be embedded in as well as . However, it’s not recommended for production use: regular Celery worker with -B parameter $ celery -A proj worker -B -l INFO The default scheduler stores data in and bases on hardcoded settings. a local celerybeat-schedule file But what if you would also like to let app admins or even users create or simply adjusts schedules? Here comes the django_celery_beat Celery allows using . Why wouldn’t we ? Because that’s precisely what this extension does without any hassle for both you and other users. custom scheduler classes keep it in a database and display conveniently in admin interface At this moment you have django_celery_beat==1.1.1 installed, . so let’s add it to INSTALLED_APPS in settings.py INSTALLED_APPS = ( ..., 'django_celery_beat', ) Django Celery Beat uses , so let it build a new table in your database by applying migrations: own model to store all schedule related data $ python manage.py migrate The last step is to To do so, you’ll need to rerun it: inform your worker to read from custom scheduler: django_celery_beat.schedulers:DatabaseScheduler. $ celery -A proj beat -l INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler ... LocalTime -> 2018-07-13 09:29:25 Configuration -> . broker -> redis://redis:6379// . loader -> celery.loaders.app.AppLoader . scheduler -> django_celery_beat.schedulers.DatabaseScheduler . logfile -> [stderr]@%DEBUG . maxinterval -> 5.00 seconds (5s) [2018-07-13 09:29:25,212: INFO/MainProcess] beat: Starting... [2018-07-13 09:29:25,212: INFO/MainProcess] Writing entries... Voila! Let’s have a look at : the admin panel All settings which were previously rooted in the code are now available at hand, thanks to what you can . Simple as that. quickly click through all the ‘boring stuff’ or outsource it to someone else Let’s add : a new periodic task Here you go! [2018-07-13 10:25:03,120: DEBUG/MainProcess] DatabaseScheduler: Fetching database schedule [2018-07-13 10:25:03,231: WARNING/ForkPoolWorker-1] Here I'm! [2018-07-13 10:25:03,232: INFO/ForkPoolWorker-1] Task my_app.tasks.send_notification succeeded in 0.0005994100356474519s: None > Although you may have an impression that configuration takes a lot of efforts, it’s just important . That’ll certainly save you from further debugging! not to miss any of initial setup steps to ensure that your Celery is sound and discovers all tasks And one more tip: if you work with a database, . Instead of that pass its primary key to get an object in its latest state straight from the database. That will help avoid situations where the object was changed and then overwritten by next task execution. don’t pass Django model objects to Celery tasks Why should you add Celery Beat to your coding routine? With the support of Celery Beat your tasks have been scheduled to execute at a specific time. To make a process even simpler for you and your users, I’ve added Django Celery Beat and a database scheduler , straight from the Django admin panel. All the tedious tasks, which bothered you, are now running smoothly under the control of Celery Beat. In the course of reading, you’ve successfully setup Celery and created your first tasks in a project. to manage your tasks without interfering with a code * FYI, a part of the title comes from the book of Al Sweigart: Automate the Boring Stuff with Python ! — I highly recommend checking it out Originally published at www.merixstudio.com .