Django asynchronous tasks without Celery

In this blog post I will guide you to implement Django asynchronous tasks without Celery. First of all I will define what I mean with the term “asynchronous task”.

What are Django asynchronous tasks?

Suppose that you want to perform a long running task in your Django web app, but you want to give an immediate response to the user without waiting for the task to finish. The task could be: sending an email, building a report, making a request to an external web service, etc. The response given to the user usually is a confirmation message that the task has started.

Every task that could take some time to complete should not block the request-response cycle between the user and your application.

If you Google for “Django asynchronous tasks” probably you’ll be directed to Celery as the way to implement asynchronous tasks with Django. Please don’t get me wrong: there is nothing wrong in Celery. Many successful projects use Celery in production with success. I also used Celery in a couple of projects, and it works well. What I find annoying is the additional effort.

Celery is yet another service to configure, launch and maintain.

Enter the uWSGI spooler

As I wrote in my other post Django – NGINX: deploy your Django project on a production server, I like to use the uWSGI application server. Well, it turns out that uWSGI provides a full featured, production ready, system to implement asynchronous tasks. They call it the uWSGI Spooler. Citing the uWSGI documentation:

The Spooler is a queue manager built into uWSGI that works like a printing/mail system.

You can enqueue massive sending of emails, image processing, video encoding, etc. and let the spooler do the hard work in background while your users get their requests served by normal workers.

A spooler works by defining a directory in which “spool files” will be written, every time the spooler find a file in its directory it will parse it and will run a specific function.

The uWSGI Spooler is very easy to configure!

Install uwsgidecorators package

To interact with uWSGI spooler from your Python code it is really convenient to install the uwsgidecorators package. Unfortunately there isn’t a recent version of uwsgidecorators on PyPI, but you can install it using apt:

sudo apt install python3-uwsgidecorators

If you use a virtualenv for your Django project (as you should) you can link the uwsgidecorator module in your virtualenv:

ln -s /usr/lib/python3/dist-packages/uwsgidecorators.py /path/to/your/virtualenv/lib/python3.6/site-packages

Create the directory for the “spool files”

First of all you have to create the directory where uWSGI will save the “spool files” used by the spooler:

mkdir -p /home/ubuntu/tasks
sudo chown www-data.www-data /home/ubuntu/tasks

Please make sure that the directory is writable by the user running uWSGI, that is www-data in the example.

Create a tasks module in your Django application

Assuming that you have a Django app named app1 you should create a module named tasks.py in the app directory. The content of the module should be something like this:

import logging
try:
    from uwsgidecorators import spool
except:
    def spool(func):
        def func_wrapper(**arguments):
            return func(arguments)
        return func_wrapper

import django
django.setup()

from .models import Model1

logger = logging.getLogger(__name__)

@spool
def long_running_task(arguments):
    id = arguments['id']
    obj1 = Model1.objects.get(pk=id)
    obj1.long_running_model_method()

from django.shortcuts import redirect
from django.contrib import messages

from .tasks import long_running_task

def async_view(request, id):
    long_running_task(id=id)
    messages.add_message(request, messages.SUCCESS,
        'Task started correctly')
    return redirect('some_other_view')

[uwsgi]
chdir = /home/ubuntu/django_project # customize with your django installation directory
env = DJANGO_SETTINGS_MODULE=project.settings.production # customize with your settings module
wsgi-file = project/wsgi.py # customize with the relative path to your wsgi.py file
workers = 1
spooler-chdir = /home/ubuntu/django_project # customize with your django installation directory
spooler = /home/ubuntu/tasks/
import = app1.tasks

service uwsgi restart

Mon Nov 18 14:50:45 2019 - [spooler /home/ubuntu/tasks pid: 8646] managing request uwsgi_spoolfile_on_www.example.com_8647_3_1383635828_1574067630_419736 ...