Django Application With Celery-

Introduction:

In modern web applications, handling asynchronous tasks efficiently is crucial for performance and user experience. Celery, a powerful distributed task queue, offers an excellent solution for managing background tasks and scheduling periodic tasks. This guide will walk you through setting up Celery with Django, using Redis as a message broker, and leveraging Celery Beat for task scheduling.

Why Use Celery?

Imagine needing to send multiple emails to users via your application. Doing this synchronously can slow down the response time, leading to a poor user experience. Celery allows you to handle such tasks asynchronously, freeing up your web application to respond to users more quickly.

Celery Concepts:

Celery:

Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system. It’s a task queue with focus on real-time processing, while also supporting task scheduling. Celery requires a solution to send and receive messages; usually this comes in the form of a separate service called a message broker. ex: Redis, RabitMq are the message brokers. Celery is task queue for executing work outside a python web application HTTP request response lifecycle. Celery is a python Package. so its can be installed with Pip.

Celery Use:

If you need to send a 10 emails to the application user. If send it through the application means it takes more time. celery worker is used to that task(send email) in background. so it take less time to respond to the user. Reduce extra overload in django application., django application will be fast. Improving user experience.It is a asynchrounus process.

Celery Process With Django:

we need medium to allocate a task to celery worker from django application. so, Redis act as a broker.Broker maintain all the queue(FIFO). We can allocate task to Multiple workers. we can scheule task on speicfc time. periodic task can be scheduled using celery beat.

Celery Beat:

It is used for scheduled or periodic task. celery beat track all the tasks in database. Broker sends the task to clery worker,.That specific time celery worker is execute that task.

Celery Beat Process:

Celery beat sends a task to broker. Broker Sends a task to worker.Worker send a request to api and retrive the response.

Celery Workers:

Celery worker process can executed multiple task at a single time.

Celery Commands:

1.Install Celery

 pip install celery

2. Run the Celery Worker server

celery -A tasks worker --loglevel=INFO

3. List all the Celery Worker Command

celery worker --help

4 . List all Celery Commands

celery --help

Implement Celery In Django

Requirements:

Python ( https://www.python.org/downloads/)
Redis(https://github.com/tporadowski/redis/releases)
Celery(pip package)
Django Project: (Set up a Django project if you haven't already)

First, we need to setup a Django Project.

Implement a Celery With in the Django Application

1.Install a Celery in your django project.

 pip install celery

2. Install the redis on your system. Redis act as a message broker between the django and celery.

Redis acts as a message broker between Django and Celery. You can download and install Redis from the following link:

https://github.com/tporadowski/redis/releases

3 . Then Install a Redis on your django project.

 pip install redis

4. Configure Celery in Django.

Add the Celery Settings to the settings.py

CELERY_BROKER_URL = "redis://127.0.0.1:6379" 
CELERY_ACCEPT_CONTENT = ["application/json"] 
CELERY_RESULT_SERIALIZER = "json" 
CELERY_TASK_SERIALIZER = "json" 
CELERY_TIMEZONE = "Asia/Kolkata"

5. Create celery.py in the django Project folder.

This file is a entrypoint for the celery within the django application. It is used for creating a task and managing workers.

6. In your Django project folder, create a celery.py file with the following content

from __future__ import absolute_import, unicode_literals 
import os 
from celery import Celery 
from django.conf import settings 
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "server.settings") 
app = Celery("server") 
app.conf.update(timezone="Asia/Kolkata") 
app.config_from_object(settings, namespace="CELERY") 
app.autodiscover_tasks() 

@app.task(bind=True) 
def debug_task(self):  
    print(f"Request:{self.request!r}")

7. Create tasks.py in your app directory

Here create a functions for allocate a work to celery

8. Add the Below code to the tasks.py

from celery import shared_task 
@shared_task(bind=True) 
def test_func(self):  
	for i in range(10):    
		print(i)  
	return "Done"

9. Import a task in the appfolder/views.py file and call that task function with in any function of the views.py

 from django.shortcuts import render
 from .tasks import test_func 
 from django.http import HttpResponse 
 def test(request):  
	test_func.delay()  
    return HttpResponse("Done")

10. Install a redis within the django application.

 pip install redis

11. Add the urls.py in the project app directory.

12. Add the below code to the urls.py in the app directory.

 from django.urls import path
 from django.urls.conf import include
 from .views import *
 urlpatterns = [  path("test", test, name="test"), ]

13. Install celery results module in python application

 pip install django-celery-results

14. Add the below line in the settings.py

This is used for stored all the task information in the django database.

CELERY_RESULT_BACKEND = "django-db"

15. Add the installed app names in the settings.py Installed Apps folder.

INSTALLED_APPS = [  "api",  "django_celery_results", ]

16. Migrate a Database

python manage.py migrate

17. We need to add below code to the __init__.py in the project root directory.

 from .celery import app as celery_app 
 __all__ = ("celery_app",)

18. Run the Celery worker Command:

It is used for monitorng a status of the task.

celery -A server.celery worker --pool=solo -l info

Note: Replace server(its my projectname) with the actual name of your Django project.

For example, if your project is named myproject, the command would be:

 celery -A myproject.celery worker --pool=solo -l info

19.Run the Django application simultaneously in django project.

python manage.py runserver

20. To test your Celery setup, call the /api/test endpoint in your browser.This will execute the Celery process.

Note: Make sure to define your route in your Django application's urls.py file

i) Browser Output

ii) Celery Terminal Output

22 . Create a Superuser from Django.

23 .Login to the Django adminpanel

https://127.0.0.1:8000/admin

24. Here, we can see our task results. every task results are stored in this task_results table.

Scheduled Task(Celery Beat) in Django Application

Celery Beat:

Celery beat is used to scheduled a task on a periodic time. Celery beat sends a task to broker. Broker Sends a task to celery worker. We can use a celery beat for sending emails on the particular time.

1. Add a Multiple User from django Admin Panel.

Here, you can add any user for the django Database. You can send a email to registered users.

2.Add your email configuration to settings.py

EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend"
EMAIL_HOST = "smtp.gmail.com" 
EMAIL_HOST_USER = #here add your email id 
EMAIL_HOST_PASSWORD = # here add your emailapp password 
EMAIL_PORT = 587 
EMAIL_USE_TLS = True

3 . Add a function in the tasks.py in the app folder

In tasks.py, add a task to send emails.

from django.contrib.auth import get_user_model 
from django.core.mail import send_mail as django_send_mail
from django.conf import settings

@shared_task(bind=True)
def send_mail(self):
    users = get_user_model().objects.all()
    print("use", users)
    for user in users:
        subject = "Hi! Celery Testing"
        msg = "message from django application"
        to_email = user.email
        django_send_mail(
            subject,
            msg,
            settings.EMAIL_HOST_USER,
            [to_email],
            fail_silently=True,
        )
    return "Success"

4 . Add a Celery Beat Settings in the Celery.py

from celery.schedules import crontab
app.conf.beat_schedule = {  
"send-mail-to-user":{   
                    "task": "api.tasks.send_mail", # here mention a app name in the task    
                    "schedule": crontab(hour=19, minute=0), # here mention a scheduled task time  
} }

This is for statically send email to the users. It will send email to the user on 7pm every day.

We can also mention the day_of_week,month in the schedule time.

month_of_year=#here mention the month no( 1 to 12), 
day_of_month= # here mention the days ( 1 to 31) 
day_of_week= #mention the week day,( 1 to 7)

5. Add the below code in the views.py

It is used for call the email scheduler function in the celery.

 from .tasks import test_func, send_mail 
 def sendmail_func():  
     send_mail.delay()  
     return HttpResponse("Mail Sent Successfully")

6 . Add the End Point in the urls.py in the app directory.

from django.urls import path 
from django.urls.conf import include 
from .views import * 
urlpatterns = [ 
        path("sendmail", send_mail, name="test"), 
]

7 . Run a Celery Beat Command in one terminal

    celery -A projectname beat -l INFO

8. Run the Django Application in another terminal

    python manage.py runserver

9. Celery beat is send task on the scheduled time.

10 . Email is Received to the all users in the database.

Conclusion

By integrating Celery with Django, you can significantly enhance your application's performance and user experience. This guide covered the basics of setting up Celery, configuring a message broker, creating and scheduling tasks, and running Celery Beat for periodic tasks. Start leveraging the power of Celery in your Django projects today!

Celery With Django