Celery

Celery is an asynchronous task queue for Python that allows you to execute tasks in the background, outside the main application flow. It is widely used for handling long-running or resource-intensive tasks without blocking the main application.

?? Key Features of Celery

? Asynchronous Processing – Runs tasks in the background without delaying user requests. ? Distributed Execution – Can run tasks across multiple worker machines. ? Scalability – Supports parallel processing to handle large workloads. ? Scheduled Tasks – Supports periodic task execution with celery-beat. ? Retry Mechanism – Can automatically retry failed tasks.

??? How Celery Works

Celery follows a producer-consumer model with three main components:

1. Task Producer (Your Application) Defines and sends tasks to the message broker.
2. Message Broker (Redis / RabbitMQ / SQS, etc.) Queues and distributes tasks to workers.
3. Worker (Celery Workers) Listens to the broker and executes tasks asynchronously.

?? Basic Setup in Django

1?? Install Celery and a message broker (Redis)

pip install celery redis
        

2?? Create a Celery instance (in project/celery.py)

from celery import Celery

app = Celery('project_name', broker='redis://localhost:6379/0')

@app.task
def add(x, y):
    return x + y
        

3?? Run the Celery worker

celery -A project_name worker --loglevel=info
        

4?? Call the task asynchronously

add.delay(10, 20)  # Executes in the background
        

?? When to Use Celery?

• Sending emails
• Processing large datasets
• Generating reports
• Scheduling periodic tasks (e.g., daily data backups)
• Web scraping