Guguweb - django

Django Rest Framework authentication: the easy way

2022-01-23T18:05:09+00:00

In this tutorial you'll learn how to implement Django Rest Framework authentication in your web application by leveraging the built-in Django session framework.

This approach is way simpler (and secure) than other popular methods such as JWT, and has only one requirement: your frontend (think Vue.js, React, ...) and your backend should be served by the same domain.

I created an example project on GitHub you can use to follow along with this tutorial.

Table of Contents

Why you should avoid JWT for Django Rest Framework authentication
Django Rest Framework settings
Django Rest Framework authentication endpoint
- Test authentication using HTTPie
A new endpoint to retrieve the user profile
- Test the user profile endpoint using HTTPie
Conclusions

Why you should avoid JWT for Django Rest Framework authentication

JWT (Json Web Token) is a very popular method to provide authentication in APIs. If you are developing a modern web application with Vue.js or React as the frontend and Django Rest Framework as the backend, there is an high probability that you are considering JWT as the best method to implement authentication.

The reality is that JWT is just one method, and unfortunately not the simpler, nor the most reliable. JWT is not supported out-of-the-box in Django Rest Framework and requires additional libraries and additional configuration for your project.

Also, implementing JWT in a secure way is quite challenging, and this is due to its complex design. Quoting the words of James Bennet, a long time Django project contributor:

"JWT is over-complex, puts too much power in the attacker's hands, has too many configuration knobs, and makes poor cryptographic choices. This is why we see vulnerabilities in JWT libraries and JWT-using systems again and again and again."

Here is some examples of JWT vulnerabilities found in the wild.

The good news is that if you can control the domain of both your backend and your frontend, you can use a much simpler method: Django sessions.

Django Rest Framework settings

Django Rest Framework comes with built-in session based authentication. To use it you have to add this in your Django settings module:

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': [
        'rest_framework.authentication.SessionAuthentication',
    ],
    'DEFAULT_PERMISSION_CLASSES': [
        'rest_framework.permissions.IsAuthenticated',
    ],
}

Remember that authentication deals with recognizing the users that are connecting to your API, while permissions deals with giving access to some resources to the users.

In the DEFAULT_AUTHENTICATION_CLASSES list you are configuring only the good old Django sessions to authenticate users. In the DEFAULT_PERMISSION_CLASSES list you are requiring that only authenticated users will access your endpoints.

Where is my token?

Django sessions are based by default on a session cookie stored on the client. There's no need for a "Token", an "Authorization" header or something like that.

If you can store that session cookie on the client and send it on every request to your API you will authenticate the user.

Django Rest Framework authentication endpoint

Now it's time to write a very simple view to let the users authenticate with a username/password.

We'll need a serializer that will take the username and password from the request and will perform the actual authentication using Django authenication framework.

Create a serializers.py file in your app and add this code to it:

from django.contrib.auth import authenticate

from rest_framework import serializers

class LoginSerializer(serializers.Serializer):
    """
    This serializer defines two fields for authentication:
      * username
      * password.
    It will try to authenticate the user with when validated.
    """
    username = serializers.CharField(
        label="Username",
        write_only=True
    )
    password = serializers.CharField(
        label="Password",
        # This will be used when the DRF browsable API is enabled
        style={'input_type': 'password'},
        trim_whitespace=False,
        write_only=True
    )

    def validate(self, attrs):
        # Take username and password from request
        username = attrs.get('username')
        password = attrs.get('password')

        if username and password:
            # Try to authenticate the user using Django auth framework.
            user = authenticate(request=self.context.get('request'),
                                username=username, password=password)
            if not user:
                # If we don't have a regular user, raise a ValidationError
                msg = 'Access denied: wrong username or password.'
                raise serializers.ValidationError(msg, code='authorization')
        else:
            msg = 'Both "username" and "password" are required.'
            raise serializers.ValidationError(msg, code='authorization')
        # We have a valid user, put it in the serializer's validated_data.
        # It will be used in the view.
        attrs['user'] = user
        return attrs

Then we can use this serializer in a login view. Add this to your views.py file:

from rest_framework import permissions
from rest_framework import views
from rest_framework.response import Response

from . import serializers

class LoginView(views.APIView):
    # This view should be accessible also for unauthenticated users.
    permission_classes = (permissions.AllowAny,)

    def post(self, request, format=None):
        serializer = serializers.LoginSerializer(data=self.request.data,
            context={ 'request': self.request })
        serializer.is_valid(raise_exception=True)
        user = serializer.validated_data['user']
        login(request, user)
        return Response(None, status=status.HTTP_202_ACCEPTED)

Mount your view in the project urls.py:

from django.contrib import admin
from django.urls import path

from users import views

urlpatterns = [
    path('admin/', admin.site.urls),
    path('login/', views.LoginView.as_view()),
]

Finally you can migrate the database, create a test user and launch the Django development server:

$ ./manage.py migrate
$ ./manage.py createsuperuser
(... follow instructions ...)
$ ./manage.py runserver

Test authentication using HTTPie

I really like HTTPie command line HTTP client. Maybe because it is written in Python and based on Python requests. :)

You can install it using

$ pip install httpie

It is one of the few packages that I like to install system wide.

After installing HTTPie, let's test a wrong login:

$ http POST :8000/login/ username=guguweb password=wrongpassword
HTTP/1.1 400 Bad Request
( ... some other response headers ... )

{
    "non_field_errors": [
        "Access denied: wrong username or password."
    ]
}

As you can see the error message is returned and the user is not logged in.

Now let's try with the right credentials:

$ http POST :8000/login/ username=guguweb password=mysecret
HTTP/1.1 202 Accepted
( ... some other response headers ... )
Set-Cookie: csrftoken=ALo4uq(...)lLoayts; expires=Tue, 24 Jan 2023 11:00:26 GMT; Max-Age=31449600; Path=/; SameSite=Lax
Set-Cookie: sessionid=l5p5zqjkhuijhjdb84; expires=Tue, 08 Feb 2022 11:00:26 GMT; HttpOnly; Max-Age=1209600; Path=/;

The user is correctly logged in and a session cookie named sessionid has been returned to our client.

If we will persist that session cookie in each request, our user will be persistently authenticated.

A new endpoint to retrieve the user profile

Let's write a view that will let the user to retrieve his user profile. Of course this view will require an already authenticated user.

First of all we need a new serializer that will be used to return the profile information. Add this to the serializers.py file introduced earlier:

from django.contrib.auth.models import User

class UserSerializer(serializers.ModelSerializer):

    class Meta:
        model = User
        fields = [
            'username',
            'email',
            'first_name',
            'last_name',
        ]

Now you can add a view to retrieve the user profile. Remember that we added rest_framework.permissions.IsAuthenticated in our DEFAULT_PERMISSION_CLASSES setting, so each view will require that the user is authenticated, unless otherwise specified.

Add this to your views.py file:

class ProfileView(generics.RetrieveAPIView):
    serializer_class = serializers.UserSerializer

    def get_object(self):
        return self.request.user

And mount the view to an endpoint in your urls.py file:

urlpatterns = [
    ...
    path('profile/', views.ProfileView.as_view()),
]

Test the user profile endpoint using HTTPie

Let's test a unauthenticated call to our new endpoint:

$ http GET :8000/profile/
HTTP/1.1 403 Forbidden
( ... some other response headers ... )

{
    "detail": "Authentication credentials were not provided."
}

The endpoint is correctly protected from unauthenticated requests. Now let's try to add an header with our session cookie.

Where I can find the session cookie value?

You can copy the sessionid value from the previous login request, it was something like this Set-Cookie: sessionid=l5p5zqjkhuijhjdb84;

$ http GET :8000/profile/ Cookie:sessionid=l5p5zqjkhuijhjdb84
HTTP/1.1 200 OK
( ... some other response headers ... )

{
    "email": "augusto@guguweb.com",
    "first_name": "Augusto",
    "last_name": "Destrero",
    "username": "guguweb"
}

Conclusions

In this tutorial you learned how to implement Django Rest Framework authentication in your web application by leveraging the built-in Django session framework.

This approach assumes that you can control the domain of both your frontend and your backend, and the two layers of the web application can be served from the same top level domain. If this is not the case, you cannot rely on browser cookies to persist the user session between subsequent requests.

In many cases this assumption is reasonable and this approach is way simper to configure and implement with respect to other token based approaches.

Don't forget to clone/fork my example project on GitHub to actually see this project running.

In a follow up post I will show you how to implement a simple Vue.js based application that will connect to the REST API implemented in this tutorial.

Have fun with Django Rest Framework!

Integrate Axios with Django Rest Framework

2021-06-12T12:34:09+00:00

Do you need to integrate the Axios HTTP client with Django Rest Framework? Then make sure to correctly configure the Django built-in Cross Site Request Forgery protection.

TLDR: add these variables to your Django settings:

CSRF_COOKIE_NAME = 'XSRF-TOKEN'
CSRF_HEADER_NAME = 'HTTP_X_XSRF_TOKEN'

Axios has built-in support for CSRF protection, and this is the default configuration:

// name of the cookie to use as a value for xsrf token
xsrfCookieName: 'XSRF-TOKEN'

// name of the http header that carries the xsrf token value
xsrfHeaderName: 'X-XSRF-TOKEN'

According to Django documentation:

"As with other HTTP headers in request.META, the header name received from the server is normalized by converting all characters to uppercase, replacing any hyphens with underscores, and adding an 'HTTP_' prefix to the name. For example, if your client sends a 'X-XSRF-TOKEN' header, the setting should be 'HTTP_X_XSRF_TOKEN'."

How to accept Paypal payments on your Django application

2021-01-12T12:34:09+00:00

In this tutorial you’ll learn how to integrate Django and Paypal to accept payments in your Django web application.

Paypal is a popular solution for online payments. Even if nowadays there are many valid alternatives, Paypal is still a big player in the market with a solid reputation and it’s trusted by millions of users.

Here is a simple step-by-step guide on how to integrate the django-paypal third party app on your website.

Table of Contents

1. Install the django-paypal app
2. Insert the app in your Django settings
3. Insert a Paypal form on a view
4. Provide an URL for Paypal IPN
5. Create views for success and failure of Paypal checkout
6. Setup a listener to detect successful Paypal payments

1. Install the django-paypal app

pip install django-paypal

2. Insert the app in your Django settings

INSTALLED_APPS = (
  ...
  'paypal.standard.ipn',
)

3. Insert a Paypal form on a view

Assuming you are using Django class-based views, you can use a FormView like this:

from django.views.generic import FormView
from django.urls import reverse
from paypal.standard.forms import PayPalPaymentsForm

class PaypalFormView(FormView):
    template_name = 'paypal_form.html'
    form_class = PayPalPaymentsForm

    def get_initial(self):
        return {
            'business': 'your-paypal-business-address@example.com',
            'amount': 20,
            'currency_code': 'EUR',
            'item_name': 'Example item',
            'invoice': 1234,
            'notify_url': self.request.build_absolute_uri(reverse('paypal-ipn')),
            'return_url': self.request.build_absolute_uri(reverse('paypal-return')),
            'cancel_return': self.request.build_absolute_uri(reverse('paypal-cancel')),
            'lc': 'EN',
            'no_shipping': '1',
        }

This is a regular FormView and the template paypal_form.html is a standard Django template like this:

<html>
  <body>
    {{ form.render }}
  </body>
</html>

Notice how we are settings some initial values for the Paypal form. Let us see all the parameters used here:

business: this is your Paypal account email. Payments will be sent to this account, so choose it carefully.
amount: the payment amount.
currency_code: the currency used in the previous amount parameter. Here is a list of accepted currency codes.
item_name: the name of the item that the customer is paying. Choose it carefully, because Paypal will show this name to the user in the payment confirmation page.
invoice: this is an “invoice code” that you can use to match the payment with an object on your database. I suggest to use some sort of primary key for your transaction here.
notify_url: a complete URI where Paypal will send an IPN (Instant Payment Notification). This is a Django endpoint you’ll have to provide and that we’ll see in a moment. Here you’ll receive an HTTP request by the Paypal servers, when the payment will be correctly registered by Paypal.
return_url: a complete URI where the customer will be redirected upon a successful payment.
cancel_return: a complete URI where the customer will be redirected when the payment is cancelled.
lc: The locale code to use in the checkout page. Here is a list of supported locale codes.
no_shipping: Can be set to 1 if you are selling digital goods, that doesn’t require shipping.

Some of this data will likely be dynamic, because it depends on the object you are selling.

4. Provide an URL for Paypal IPN

In your Django urls.py file, add an urlconf like this:

path('paypal/', include('paypal.standard.ipn.urls')),

5. Create views for success and failure of Paypal checkout

You can add two TemplateView to show your user a message upon success or failure of the Paypal checkout.

from django.views.generic import TemplateView

class PaypalReturnView(TemplateView):
    template_name = 'paypal_success.html'

class PaypalCancelView(TemplateView):
    template_name = 'paypal_cancel.html'

Then you can mount these views in your urls.py:

from . import views
urlpatterns = [
    path('/paypal-return/', views.PaypalReturnView.as_view(), name='paypal-return'),
    path('/paypal-cancel/', views.PaypalCancelView.as_view(), name='paypal-cancel'),
    ...
]

The names of the urls should match with those defined in the form initial data, as shown at step 3.

6. Setup a listener to detect successful Paypal payments

When a successful payment if performed on Paypal, you’ll receive a so-called IPN: “Instant Payment Notification” from Paypal.

django-paypal uses the Django signal dispatcher to let you hook your code when a successful payment arrives. You can define a listener like this:

from paypal.standard.models import ST_PP_COMPLETED
from paypal.standard.ipn.signals import valid_ipn_received

@receiver(valid_ipn_received)
def paypal_payment_received(sender, **kwargs):
    ipn_obj = sender
    if ipn_obj.payment_status == ST_PP_COMPLETED:
        # WARNING !
        # Check that the receiver email is the same we previously
        # set on the `business` field. (The user could tamper with
        # that fields on the payment form before it goes to PayPal)
        if ipn_obj.receiver_email != 'your-paypal-business-address@example.com':
            # Not a valid payment
            return

        # ALSO: for the same reason, you need to check the amount
        # received, `custom` etc. are all what you expect or what
        # is allowed.
        try:
            my_pk = ipn_obj.invoice
            mytransaction = MyTransaction.objects.get(pk=my_pk)
            assert ipn_obj.mc_gross == mytransaction.amount and ipn_obj.mc_currency == 'EUR'
        except Exception:
            logger.exception('Paypal ipn_obj data not valid!')
        else:
            mytransaction.paid = True
            mytransaction.save()
    else:
        logger.debug('Paypal payment status not completed: %s' % ipn_obj.payment_status)

Here we assumed that you had a model MyTransaction on your Django application, and that you want to set the transaction as paid when the Paypal payment is confirmed.

That’s it! Following these steps you should have a working Paypal integration in your Django project. Please let me know if you have questions or comments.

Django on a production server

2020-05-08T10:34:43+00:00

Is your Django project ready for production? Follow these guides to bring your Django project on a production server!

There are many good tutorials out there for learning Django and developing projects locally. But I think that simple tutorials for production deployment is somewhat lacking. In this guide I'll try to fill this gap a little bit. 😉

In the last months I wrote some tutorials that will show you how to bring your Django project on production. Here is a summary of each tutorial.

1. Django NGINX: deploy your Django project on a production server

This tutorial will give you the basics on deployment using the NGINX webserver and uWSGI application server.

Django NGINX: deploy your Django project on a production server

2. How to deploy a Django project in 15 minutes with Ansible

This tutorial will make a step further and will show you how to leverage the power of Ansible to automate all the steps needed for a production deployment.

How to deploy a Django project in 15 minutes with Ansible

3. Bitbucket Pipelines and Ansible: Continuous delivery for your Django project

If you are using Bitbucket you can automate the deployment when you push some code on your repository.

Bitbucket Pipelines and Ansible: Continuous delivery for your Django project

4. Django asynchronous tasks without Celery

Did you know that uWSGI provides a production ready task queue? In this tutorial I explain how to configure and use it for your Django asyncronous tasks.

Django asynchronous tasks without Celery

Remote debugging a Django project in VS Code

2020-04-20T08:02:16+00:00

In this tutorial you will learn how to use VS Code for remote debugging. I’ll show you how to debug a Django application running on a remote server over SSH or in a Docker container.

Imagine the following scenario: You developed a Django application on your laptop. You did your tests and made some debug. The application worked fine. Finally you deploy the application on a production server. Something goes wrong. There is a bug to hunt down and you cannot reproduce it locally.

You could connect to the remote server using SSH, then use a remote editor like vim on nano to tweak the code. You could put some print("I'm here!") here and there, trying to find the bug.

But wait, aren’t you a VS Code user? Why you have to debug in an editor you don’t know? Wouldn’t be great if you could connect to the remote environment from VS Code and debug your application inside the IDE you love?

It turns out that VS Code has some extensions made exactly for that purpose, and you’ll learn how to use them.

At the end of this tutorial you will be able to:

Connect with VS Code to a remote server using SSH and remotely debug a web project.
Run a Docker container within VS Code and debug a project inside the container.

Before starting the tutorial let me say that I assume you are actively using Visual Studio Code for development and you already know how to install VS Code extensions and how to manage a Python project in the IDE. If you need a refresh you can check this introductory tutorial.

Table of Contents

1. Remote Debugging Over SSH
2. Remote Debugging on a Docker Container
3. Other Remote Debugging Scenarios
- WSL
- LiveShare
4. Conclusion

1. Remote Debugging Over SSH

The first scenario I will present is remote debugging on a server that is accessible via SSH. I assume that you already know how to connect to a remote server using SSH. You can find good resources on the internet if you need more information.

What if You Do Not Have a Remote SSH Server?

Of course to follow along with this part of the tutorial you’ll need an SSH remote server to connect to. If you don’t have one readily available you can spin up a local virtual machine using Vagrant.

After installing Vagrant and VirtualBox, creating a virtual machine is as simple as:

$ vagrant init hashicorp/bionic64
$ vagrant up

You’ll find an Ubuntu Server listening for SSH connections on localhost:2222. Username is vagrant, password is vagrant. Easy enough!

If you prefer to test on a real remote environment, you can create a virtual machine on Azure.

In this tutorial I’ll use the Vagrant method to spin up a test SSH “remote” server on localhost, port 2222.

Install the VS Code Extension

First of all you have to install the Remote – SSH extension from the VS Code marketplace. Go to the Extensions section in VS Code and install the extension from there:

If the extension has been installed correctly you will see a new green icon in the bottom left corner of the VS Code window.

Generate an SSH Key Pair and Configure Server for Remote Access

You’ll need to access the SSH server without entering the passphrase every time. For this you’ll need an SSH keypair to connect to the server. If you don’t already have one you can generate one by typing:

ssh-keygen -t rsa -b 2048

You can leave the SSH key passphrase empty, when prompted. In this way the public key will be saved in the .ssh/id_rsa.pub file under your home directory.

The last step is copying the public key on the remote server and appending it to the .ssh/authorized_keys file, inside the home directory of the remote user you’ll use to connect.

If you are using the test SSH server created with Vagrant, as shown previously, the commands to copy the SSH public key in the right place are:

localhost:~$ scp -P 2222 ~/.ssh/id_rsa.pub vagrant@localhost:
localhost:~$ ssh -p 2222 vagrant@localhost
vagrant@vagrant:~$ cat id_rsa.pub &gt;&gt; ~/.ssh/authorized_keys

When prompted for a password remember that both the username and the password are vagrant.

The last command is ran on the remote SSH server. You should now be able to login to SSH server without the need to write your passphrase.

Connect to Remote Server in VS Code

It’s time to connect to the remote server in VS Code. Click on the green icon on the bottom left corner:

A command prompt will appear at the top of the windows, click on Remote-SSH: Connect to Host….

If you already have some SSH servers configured in your .ssh/config file you’ll see them listed here. In our example we will connect to a new SSH server, by writing the connection parameters in the user@host:port form.

Press Enter and a new VS Code windows will appear. That’s literally your window to the remote system.

The file explorer will let you to add remote directories in your workspace. You’ll be able to edit files as if they were local.

You can even open a terminal inside VS Code and it will act as a remote terminal. You’ll make use of the remote terminal in the following section.

Create a Simple Django Project on Remote Server

A Django project will serve just as an example of what you can do with VS Code SSH extension to debug your remote application. Even if you’re not familiar with Django you’ll be able to follow along with the tutorial. If otherwise you’d like to better understand what you are doing here you can read this introductory Django tutorial.

In the VS Code remote window, open a terminal using the “Terminal > New terminal” menu. Notice how this terminal is running on the remote server, and not locally on your machine.

First you’ll need to install a package on the remote server. This is needed to create a virtualenv for the project:

vagrant@vagrant:~$ sudo apt update
vagrant@vagrant:~$ sudo apt install python3-venv

Then you can create a virtualenv and activate it:

vagrant@vagrant:~$ python3 -mvenv vscode-example-env
vagrant@vagrant:~$ . vscode-example-env/bin/activate

Now install Django on the virtualenv and create a new project:

(vscode-example-env) vagrant@vagrant:~$ pip install django==2.2.12
(vscode-example-env) vagrant@vagrant:~$ django-admin startproject vscode_example

Mapping Ports to Test the Project

Now that you’ve created the example Django project, go into the project directory, create/migrate the Django database and run the Django development server:

(vscode-example-env) vagrant@vagrant:~$ cd vscode_example
(vscode-example-env) vagrant@vagrant:~$ ./manage.py migrate
(vscode-example-env) vagrant@vagrant:~$ ./manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
April 01, 2020 - 17:35:01
Django version 2.2.12, using settings 'vscode_example.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

As you can see from the command output the Django development server is listening on http://127.0.0.1:8000/.

Note: Don’t confuse the http://127.0.0.1:8000/ you see here. That is the localhost of the server and not the local address of your PC.

That address will not be accessible from your PC because it’s local to the server. Fortunately VS Code lets you map ports of remote server to ports of your PC. This will let you access server ports as if they were local.

Click on VS Code “Remote explorer” left pane:

You’ll see that in the “Forwarded ports” section there is a port on the server that is “”Not Forwarded”. Hover with your mouse on the port and click on the “+” icon:

Now that the server 8000 port is mapped on your PC 8000 port, you’ll be able to access your Django project on http://localhost:8000/:

Note: Don’t forget to forward the 8000 port every time you restart the Django development server.

A buggy Django app

Now that you have a Django project running remotely why don’t you create a Django app to test some remote debugging? Let’s do it!

Back to the VS Code terminal and stop the Django development server with CTRL-C. Then create a Django app:

(vscode-example-env) vagrant@vagrant:~$ ./manage.py startapp todo

Every example project has a todo app, isn’t it?

Now it’s time to edit some files remotely using VS Code. As promised you won’t use vim or nano over SSH!

Click on the VS Code file explorer pane, then on Open Folder button:

From the menu choose the directory you want to add to VS Code workspace, that is vscode_example in our case:

Select the directory and click on “OK”. Now you can browse the remote files in the VS Code file explorer!

Open the todo/view.py file and add the following code:

from django.http import HttpResponse

from random import choice

def random_todo(request):
    todo_list = [
        'Buy milk',
        'Clean the room'
        'Write tutorial',
        'Read books',
    ]
    return HttpResponse(choice(todo_list))

Then open the vscode_example/urls.py file and edit like this to add a route to the view:

from django.contrib import admin
from django.urls import path

import todo.views

urlpatterns = [
    path('admin/', admin.site.urls),
    path('todo/random/', todo.views.random_todo),
]

Now you can return to the terminal and restart the Django development server:

(vscode-example-env) vagrant@vagrant:~$ ./manage.py runserver

Check that the virtualenv is active before running the server. Also check that the port 8000 is mapped in the VS Code “Remote explorer” left pane, as seen previously.

Open your web browser and go to the newly created url at http://localhost:8000/todo/random/. You should see a randomly chosen todo item. Try to refresh the page a couple of times, and then a bug will show up:

There is something wrong in our todo list! Of course you can spot the bug quite easily in our toy project, but why don’t leverage a full featured debugger in VS Code to check what’s going on?

Debugging in VS Code

Stop the Django development server with CTRL-C and click on the VS Code “”Run” left pane. There you’ll see a create a launch.json file link.

Click on the link and select More… from the menu. You can then install the Python extension on remote server by clicking on Install in SSH: localhost.

The remote extension will install and you’ll have to reload the VS Code window to activate it.

Go back to the Run left pane and click on create a launch.json file link again. This time you’ll see a Django item in the menu. Click on it.

A launch.json file will open in VS Code. You’ll only have to add the path to your virtualenv, by adding a pythonPath entry in the configuration.

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more info, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Django",
            "type": "python",
            "request": "launch",
            "pythonPath": "/home/vagrant/vscode-example-env/bin/python",
            "program": "${workspaceFolder}/manage.py",
            "args": [
                "runserver",
                "--noreload"
            ],
            "django": true
        }
    ]
}

Now you can run the project for remote debugging in VS Code by hitting F5. Go to the todo/views.py file and add a breakpoint on line 13:

Then try to reload the http://localhost:8000/todo/random/ page. You’ll see that execution will block and you’ll be able to inspect variables in VS Code:

In this simple case the bug was a missing comma in the todo_list definition.

That was a trivial bug in a simple project, but imagine to leverage the power of VS Code debugger in a real project running remotely. Now you learned all the tools needed to accomplish this task and make your remote debugging easier.

2. Remote Debugging on a Docker Container

Remote debugging can be useful also on another common scenario: debugging a project while running in a Docker container.

Suppose that your team uses a Docker container to have a common and established development environment. You can leverage VS Code Remote – Containers extension to build and run the container and attach to it for remote debugging.

I assume that you have some basic knowledge on how to obtain a Docker image and run it either by using Docker command line or VS Code Docker extension. If you need more information you can refer to this tutorial.

Install Docker

You have to install Docker to follow this tutorial. If you are on Windows or Mac you can install Docker Desktop. If you are on Linux you can install Docker using your distribution package manager.

Install the VS Code Extension

The next step is to install the Remote – Containers extension from the VS Code marketplace. Go to the Extensions section in VS Code and install the extension from there:

If the extension has been installed correctly you will see a new green icon in the bottom left corner of the VS Code window.

Clicking on the green icon will bring up the Remote - Containers commands.

An Example Container for Your Project

In this tutorial you’ll clone an example project container from a Git repository. That container uses the same example Django application seen in previous section of this tutorial. It will serve as a starting point to see how you can build, run and perform remote debugging on a Python project running in a Docker container.

Clone the repository using Git:

git clone https://github.com/baxeico/vscode_django_container.git

Go into the vscodedjangocontainer directory and launch VS Code:

$ cd vscode_django_container/
$ code .

VS Code will notice that this directory contains a configuration for remote containers and it will prompt you to reopen the project in a container:

Click on the Reopen in Container button and VS Code will build the container and configure the project for remote debugging.

Understanding devcontainer.json

The Remote Containers extension will look for a file called devcontainer.json in the .devcontainer directory.

The content of the file will look like this:

{
    "name": "Django container example",
    "build": {
        "dockerfile": "Dockerfile",
        "context": "..",
    },

    // Set *default* container specific settings.json values.
    "settings": {
        "terminal.integrated.shell.linux": "/bin/bash",
        "python.pythonPath": "/usr/local/bin/python3",
    },

    // IDs of extensions you want installed when the container is created.
    "extensions": [
        "ms-python.python"
    ],

    // make a list of ports inside the container available locally.
    "forwardPorts": [8000],

    // run commands after the container is created.
    "postCreateCommand": "pip3 install -r requirements.txt; cd vscode_example; ./manage.py migrate",
}

In the build section you can define the parameters used to build the container. In this case you are using a Dockerfile contained in the same directory. The Dockerfile uses the Python3 official image from Docker Hub, without modifications.

In the settings section you can define some specific settings for VS Code, when running in the container. Notice how the pythonPath option is set to the python3 executable path in the Docker image.

The extensions section will let you define the VS Code extensions you want to install in the container. The ms-python.python is required to let you debug Python applications.

forwardPorts is useful to let you access your application as if it was local, by exposing container ports on the local host. Here you are mapping the 8000 port of the container to the same port of your local host.

postCreateCommand is used to install the requirements of the Python application using pip and to migrate the Django database after the container has been created.

Here you can find a complete devcontainer.json reference.

Debugging in VS Code

Clicking on F5 will start the Django application. You can open the browser and point it to http://localhost:8000/ to see the app running.

VS Code will let you debug the code, set breakpoints, inspect variables as if the project is running locally.

As usual, the launch configuration is made in the launch.json file, inside the .vscode directory. If you look at the example inside the container it will launch the Django application on port 8000.

This tutorial used a very simple Django app in a minimal Docker container. Imagine to use your own container and have your team working and remote debugging on it using VS Code.

All you have to do is to define a proper devcontainer.json for your environment and use the Docker image of your choice with your configuration.

3. Other Remote Debugging Scenarios

There are also other use cases for remote debugging that are covered by other VS Code extensions.

I’ll not go into much details here, but I think that it’s useful to know that these extensions exist.

WSL

Windows Subsystem for Linux is a feature in Windows 10 that will let you to install a complete Linux distribution (e.g. Ubuntu 18.04) in your local Windows 10 installation. The Linux and Windows installations will be isolated, so you cannot run and debug an application running in WSL directly from VS Code, if running on Windows.

You need the Remote WSL extension to remotely debug your application running in WSL, as it was local.

LiveShare

While technically this feature cannot be considered “remote debugging”, it is worth knowing that VS Code has a LiveShare feature that will let you share a debugging session with one or more colleagues.

You can think of it as a sort of “video conference” session where one developer is the host and he can share his VS Code window with other people.

Actually this feature is more powerful than a simple video conference session because for instance you’ll be able to debug the code concurrently, by inspecting different variables or files independently without having to negotiate control.

4. Conclusion

In this tutorial you learned how to:

Connect with VS Code to a remote server using SSH and remotely debug a web project.
Run a Docker container within VS Code and debug a project inside the container.

Using the correct tools you’ll never resort to wild debugging sessions over SSH using vim or nano. No more print("I'm here") to catch those nasty bugs! Now you learned how to use a full featured IDE for remote debugging.

I want to publicly thank Geir Arne Hjelle who helped me in shaping up the outline of this tutorial.

Optimize Django memory usage

2020-03-27T16:55:31+00:00

Django memory usage is usually quote good, but sometimes – if you use the Django ORM without really knowing what is doing behind the scenes – you can see a huge spike in RAM usage. Fortunately there exist some simple methods to optimize Django memory usage.

Table of Contents

1. The problem
2. A bit of Django memory profiling
3. Optimize Django memory usage: using iterator() method
4. Optimize Django memory usage: using pagination
5. Conclusion

1. The problem

Consider this is the apparently innocent view

from django.http import HttpResponse

from .models import FirstModel

# this view will make crazy use of the RAM ;)
def my_view(request):
    # this queryset contains about 100k records
    # each of them has many ForeignKeys to other models
    huge_queryset = FirstModel.objects.all()

    f = open('dumb.dump', 'w')

    for record in huge_queryset:
        print >>f, record

    f.close()
    return HttpResponse('Dumb dump completed!')

as you can see it’s very simple, the peculiarity here is the dimension of the queryset, because the table contains about 100k records, and each record has several ForeignKey fields to other models.

An experienced Django developer should immediately see the problem, but it can be interesting to analyze the problem a little bit more.

If you are impatient, go directly to my solution.

2. A bit of Django memory profiling

When you want to profile memory usage in Python you’ll find some useful tools. I choose to use objgraph. The only drawback is that objgraph is designed to work in a python console, while my code is running in a Django powered website.

So I put together some code to redirect the standard output used by objgraph to my beloved Django logging system, and here is the result.

import objgraph
import sys
import logging

from django.http import HttpResponse

from .models import FirstModel

logger = logging.getLogger(__name__)

class LoggerWriter:
    def __init__(self, logger):
        self.logger = logger

    def write(self, message):
        if message != '\n':
            self.logger.debug(message)

def my_view(request):
    sys.stdout = LoggerWriter(logger)

    huge_queryset = FirstModel.objects.all()

    f = open('dumb.dump', 'w')

    row = 0
    for record in huge_queryset:
        print >>f, record
        row += 1
        if not (row % 1000):
            objgraph.show_growth()

    f.close()
    return HttpResponse('Dumb dump completed!')

In the Django log I saw something like this (I cut from the log some internal objects not so interesting in this example):

[24/09/2014 10:35:31] DEBUG [leaky_app.views:31] dict 106524 +106524
[24/09/2014 10:35:31] DEBUG [leaky_app.views:31] ModelState 101328 +101328
[24/09/2014 10:35:31] DEBUG [leaky_app.views:31] FirstModel 98327 +98327

[24/09/2014 10:36:23] DEBUG [leaky_app.views:31] dict 109526 +3002
[24/09/2014 10:36:23] DEBUG [leaky_app.views:31] ModelState 104329 +3001
[24/09/2014 10:36:23] DEBUG [leaky_app.views:31] SecondModel 1999 +1000
[24/09/2014 10:36:23] DEBUG [leaky_app.views:31] ThirdModel 1999 +1000
[24/09/2014 10:36:23] DEBUG [leaky_app.views:31] FourthModel 2000 +1000

[24/09/2014 10:37:17] DEBUG [leaky_app.views:31] dict 112874 +3348
[24/09/2014 10:37:17] DEBUG [leaky_app.views:31] ModelState 107330 +3001
[24/09/2014 10:37:17] DEBUG [leaky_app.views:31] FourthModel 3000 +1000
[24/09/2014 10:37:17] DEBUG [leaky_app.views:31] SecondModel 2999 +1000
[24/09/2014 10:37:17] DEBUG [leaky_app.views:31] ThirdModel 2999 +1000

FirstModel has some ForeignKey fields to SecondModel, ThirdModel and so on. That means that every 1000 row three different objects are loaded in memory. But why Django is putting all those objects in memory? After all I want only to write a record in my dumb.dump file.

It turns out that Django caches the results of each queryset, when you iterate over it. Each and every object in the queryset will be saved in memory. This lets the ORM to access the queryset again very efficiently, without the need to hit the database again.

3. Optimize Django memory usage: using iterator() method

You can avoid caching and simply iterate over a queryset using the iterator method on the queryset. The Django documentation clearly states:

“For a QuerySet which returns a large number of objects that you only need to access once, this can result in better performance and a significant reduction in memory.”

So the first solution to the problem is simply adding .iterator() to the iterated queryset, like this:

from django.http import HttpResponse

from .models import FirstModel

def my_view(request):
    # this queryset contains about 100k records
    # each of them has many ForeignKeys to other models
    huge_queryset = FirstModel.objects.all().iterator()

    f = open('dumb.dump', 'w')

    for record in huge_queryset:
        print >>f, record

    f.close()
    return HttpResponse('Dumb dump completed!')

4. Optimize Django memory usage: using pagination

Even using iterator() the data is first fetched on the client side by the database driver, occupying a lot of memory. This happens because Django doesn’t yet support server side cursors.

A workaround for this issue is using an utility function like this:

import gc

def queryset_iterator(qs, batchsize = 500, gc_collect = True):
    iterator = qs.values_list('pk', flat=True).order_by('pk').distinct().iterator()
    eof = False
    while not eof:
        primary_key_buffer = []
        try:
            while len(primary_key_buffer) < batchsize:
                primary_key_buffer.append(iterator.next())
        except StopIteration:
            eof = True
        for obj in qs.filter(pk__in=primary_key_buffer).order_by('pk').iterator():
            yield obj
        if gc_collect:
            gc.collect()

You can use simply by passing a normal Django queryset as the single parameter, like this:

for obj in queryset_iterator(HugeQueryset.objects.all()):
   # do something with obj
   pass

This is useful when you want to delete an high number of objects that have a ForeignKey pointing to them. In that case using the Queryset .delete() method doesn’t help because Django has to fetch each object in memory to handle the deletion cascade policy given for the objects being deleted. You can do something like this to mitigate that effect:

for obj in queryset_iterator(HugeQueryset.objects.all()):
   obj.delete()

5. Conclusion

In this tutorial you learned how to optimize Django memory usage, especially when dealing with large querysets. Django ORM is really powerful and the abstraction it provides will let you write complex queries easily. But you have to be careful when using the ORM with large querysets because the effects on memory and CPU usage can be large as well.

Bitbucket Pipelines and Ansible: Continuous delivery for your Django project

2020-01-03T08:33:41+00:00

“Bitbucket Pipelines and Ansible: Continuous delivery for your Django project” was originally published as a guest post on Bitbucket Blog.

Table of Contents

Introduction
Let the automation begin!
Setup your pipelines
Conclusion

Introduction

Ansible is a powerful tool for the automation of your server administration. It can be used both to install and configure the software needed for your application and to deploy a new version of your application.

Bitbucket Pipelines are basically Docker containers, hosted in the Bitbucket infrastructure, that you can launch automatically to build or deploy your code. They can be attached to “events” happening in your repository, usually a push on one branch.

These tools are complementary to each other, because on Ansible you can define exactly what you want to do on your target server to deploy your application, while on Bitbucket Pipelines you can configure and launch a container to perform the actual deploy.

In a previous post I described how to deploy a Django project using Ansible. Here I explain how to make a step further and deploy a Django project using Bitbucket Pipelines and Ansible.

The use-cases for deploying a Django project automatically are many. Even if your Django project has a simple architecture, there are many steps you have to perform in the right order to complete a successful deploy in the general case. As a bare minimum you’ll have to: update the application code, install or update project dependencies, migrate the database, collect static files, restart the application server.

Automating all these steps in a reproducible and formal way will save you a lot of time in the long run and will help you to avoid human errors. Think about how many times you’ll deploy a new version of your application in the future.

If your architecture is more complex, the number of steps to perform will be larger and the deploy process will be more error-prone, so the benefits of an automated deploy will be even greater.

Let the automation begin!

You can clone or fork this Bitbucket repository to follow along with this tutorial.

First thing first, add a file named bitbucket-pipelines.yml to the root of your repository. The content of the file should be similar to this:

# use the official Python 2.7.16 docker image
image: python:2.7.16

pipelines:
  branches:
    # deploy only when pushing to the master branch
    master:
      - step:
          name: Deploy to prod
          # this is the name of the Bitbucket Deployment
          deployment: Production
          caches:
            # cache the Ansible installation
            - pip
          script:
            # install Ansible
            - pip install ansible==2.8.0
            # go into the ansible directory
            # in this same repository
            - cd ansible
            # perform the actual deploy
            - ansible-playbook -i ./hosts deploy.yaml

Let’s examine the configuration line by line:

On line 2 you define the docker image you want to use;
On line 7 you define the branch that will trigger the deploy (master in this example);
On line 11 you define the deployment, that is basically an environment (managed by Bitbucket) where you can define some variables. In this way you can have different deployments (staging, production, …) with different variables that you can use in your build/deploy steps. In this example I do not use this feature, and I merely use one of the default Bitbucket deployments (named Production) without configuring any variable in this environment;
On line 14 you are asking Bitbucket to cache the installation of packages installed by pip. This is useful to avoid to reinstall every time the Python libraries you need for your deployment (Ansible 2.8.0 in this example) from scratch. The build time for Bitbucket Pipelines is limited to 50 minutes/month in the free plan, so it’s important to have the libraries already installed to save some build/deploy time.
From line 16 to line 22 you define the actual steps for deployment: install Ansible (if not already cached), go into the ansible directory, make the deploy. Please check the playbook ansible/deploy.yaml to see what it will actually do on the target server.

Ansible will perform the actual deploy by:

pulling the master branch from your Git repository;
installing all needed production requirements;
running Django migrate and collectstatic commands;
restarting the application server uwsgi.

Please check the Ansible playbook ansible/deploy.yaml to see what it will actually do on the target server in details.

Commit and push the bitbucket-pipelines.yml file on the root of your Bitbucket repository.

Setup your pipelines

Now go on your repository settings and click Settings under the section PIPELINES:

Pipelines settings

Now click on Enable Pipelines:

Pipelines settings

If you have read my previous post on how to deploy a Django project using Ansible, you should know that Ansible work through an SSH connection made between the client host (the Bitbucket Pipeline docker container in this case) and the server where you want to deploy your code.

For this SSH connection to work you have to setup an SSH public/private keypair on the Bitbucket Pipeline, and allow the public key on your server authorized_keys.

Go on the settings of your repository, under PIPELINES navigate to SSH keys:

Pipelines settings

Now click on “Generate keys” to let Bitbucket generate a new SSH keypair for you.

Pipelines settings

Now you can copy the generated public key and paste it in the .ssh/authorized_keys file of your deploy target.

Pipelines settings

You’ll also have to add your server fingerprint to the Known hosts section in the same page. Insert your target server hostname in the “Host address” field, press the “Fetch” button and finally the “Add host” button.

Known hosts settings

You can refer to the Bitbucket documentation about SSH keys, if you need more details.

You can test that the deploy is working simply by pushing some changes to the master branch of the repository

If you want to push some changes on master, but you do not want to deploy the change automatically on the server, you can put the string [skip-ci] in your commit message.

Conclusion

In this tutorial I explained a simple yet powerful setup for the continuous delivery of a Django project. This is only a little example of what you can do with Bitbucket Pipelines and Ansible, but I hope that this helped you to understand the great potential of these technologies.

I think that learning and implementing this kind of automation in your projects is always worth the effort. Time saved and less human errors in the deploy process will make a big difference in the long run.

Automate your Telegram channel with a Django Telegram Bot

2019-12-09T17:45:14+00:00

In this post I will guide you in a very interesting and funny development task: automate your Telegram channel with a Django Telegram Bot.

Telegram is a popular instant messaging application. It’s not as widespread as Whatsapp, but its user base is constantly growing and according to Wikipedia:

The number of monthly Telegram users as of October 2019 is 300 million people worldwide

A very interesting aspect of the Telegram platform is the ability to create channels and bots to automate the broadcasting of messages to large audiences. These features open the possibility to better engage your users using Telegram, for example by sending them news, offers, etc. all from your existing Django web project! Let’s see how to do it.

Table of Contents

1. Create a Telegram channel
2. Create a Telegram Bot
3. Make the bot an Admin for your Telegram channel
4. Install python-telegram-bot
5. Edit your Django settings to add Telegram parameters
6. Send a message to the channel from your Django code
7. Conclusions
8. Do you want more power for your Django Telegram bot?

1. Create a Telegram channel

Nothing special here: you can do this from the Telegram app on your phone. Just tap “New Channel” on the Telegram main menu. Choose a nice name and a clear description.

Create the Telegram channel

Then setup the channel as Public and choose a nice “permalink” for your channel:

Setup the Telegram channel

2. Create a Telegram Bot

Now the funny part begins: search for a contact named “BotFather” on Telegram

Search for BotFather

This is a bot to manage bots, a sort of meta-bot. 🙂 Start a chat with him and write /help to know what are the available commands. To create a new bot you have to write the command /newbot.

The BotFather will ask you a couple of questions to setup your bot. Basically you’ll have to choose a “display name” and a username for your bot.

Create the Telegram bot

As you can see in the screenshot above, the BotFather will write you a token that you should save and keep secret. This token will be used to interact with your bot using the Telegram Bot API.

You can also set a description, an about text and even a nice profile picture for your bot; for example something like this. /help command is your friend to find out how you can do these things.

3. Make the bot an Admin for your Telegram channel

Your bot will be used to automatically post messages on your channel, so you have to add the bot to the group of channel’s administrators. Tap on the channel’s name and then on Administrators:

Edit the Telegram channel

Then on Add Admin:

Edit the Telegram channel administrators

Now search for the bot you created at step 2:

Add the bot as the channel admin

And add it to the channel with the minimum required permission, that is only Post Messages for the scope of this tutorial:

Setup the bot permissions

Ok, you have completed all the steps needed on the Telegram app, and from now on you’ll work only on the Python/Django side.

4. Install python-telegram-bot

To interact with the Telegram bot API I strongly recommend to use a wrapper, so you don’t have to deal with low level details. There are three different projects for Python, but I chose python-telegram-bot because it looks quite simple to integrate and use.

Install with:

pip install python-telegram-bot

Or add the project to your requirements:

tornado==5.1.1 # dependency of python-telegram-bot
python-telegram-bot==12.2.0

I also had to fix the version of tornado in my requirements, because the requirement made by python-telegram-bot is tornado>=5.1 and this installed tornado 6 in my case, which is not compatibile with my version on Python.

5. Edit your Django settings to add Telegram parameters

This is not strictly required, but I strongly suggest you to keep all the Telegram related parameters in your Django settings. Add a dictionary called TELEGRAM like this in your Django settings module:

TELEGRAM = {
    'bot_token': '123456789:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
    'channel_name': 'guguweb',
}

Of course you have to customize the bot_token parameter using the token you received from the BotFather at step 2, and the channel_name parameter with the actual name of your channel.

6. Send a message to the channel from your Django code

Suppose you have a Django powered website where you publish events for your community. An Event model could be something like this:

class Event(models.Model):
    title = models.CharField(max_length=255)
    description = models.TextField(blank=True)
    start_date = models.DateTime(null=True)

Now suppose that you want to send a message on your Telegram channel when you post a new event. To do this create a Django template that will be used to define the HTML for the Telegram message, with a content similar to this:

<a href="{{ event.url }}">{{ event.title }}</a>
{{ event.description|truncatewords:30 }}
<strong>Date:</strong>{{ event.start_date|date:'SHORT_DATE_FORMAT' }}

Consider that only a subset of HTML tags are supported by Telegram and that you cannot nest HTML tags. Save the template with the name telegram_message.html.

You can then define a function to actually create the message and send it to the Telegram bot:

import telegram # this is from python-telegram-bot package

from django.conf import settings
from django.template.loader import render_to_string

def post_event_on_telegram(event):
    message_html = render_to_string('telegram_message.html', {
        'event': event
    })
    telegram_settings = settings.TELEGRAM
    bot = telegram.Bot(token=telegram_settings['bot_token'])
    bot.send_message(chat_id="@%s" % telegram_settings['channel_name'],
                     text=message_html, parse_mode=telegram.ParseMode.HTML)

The function takes an Event instance as the only argument, uses the Django template system to render the HTML, and then uses an instance of telegram.Bot to actually send the message. So simple, isn’t it?

If you plan to call the function from one of your views in a request-response cycle, I strongly suggest you to configure an asynchronous task to send the message, like I explained in my post Django asynchronous tasks without Celery:

Django asynchronous tasks without Celery

7. Conclusions

In this post I’ve shown you how to automate your Telegram channel with a Django Telegram Bot. I think that this approach opens many interesting possibilities for better engaging the users of your Django web application.

Please don’t hesitate to ask questions or make comments in the form below or by contacting me. Of course you can also join my Telegram channel, the one that I created when writing this tutorial: https://t.me/guguweb. I’ll use the channel to post updates from this blog, so if you find this post useful please join the channel. 😉

8. Do you want more power for your Django Telegram bot?

Of course you can! In a following post I will explain how to make a step further and make your Telegram bot more useful, by giving it the power to answer inquiries coming from Telegram users.

The bot (you guessed it) will use Django as the backend to receive messages from Telegram and send replies to the users. Stay tuned!

Django asynchronous tasks without Celery

2019-11-21T10:18:45+00:00

In this blog post I will guide you to implement Django asynchronous tasks without Celery. First of all I will define what I mean with the term “asynchronous task”.

Table of Contents

What are Django asynchronous tasks?
Enter the uWSGI spooler
Install uwsgidecorators package
Create the directory for the “spool files”
Create a tasks module in your Django application
Call the asynchronous task from your Django view
Configure uWSGI to use the spooler
Conclusion

What are Django asynchronous tasks?

Suppose that you want to perform a long running task in your Django web app, but you want to give an immediate response to the user without waiting for the task to finish. The task could be: sending an email, building a report, making a request to an external web service, etc. The response given to the user usually is a confirmation message that the task has started.

Every task that could take some time to complete should not block the request-response cycle between the user and your application.

If you Google for “Django asynchronous tasks” probably you’ll be directed to Celery as the way to implement asynchronous tasks with Django. Please don’t get me wrong: there is nothing wrong in Celery. Many successful projects use Celery in production with success. I also used Celery in a couple of projects, and it works well. What I find annoying is the additional effort.

Celery is yet another service to configure, launch and maintain.

Enter the uWSGI spooler

As I wrote in my other post Django – NGINX: deploy your Django project on a production server, I like to use the uWSGI application server. Well, it turns out that uWSGI provides a full featured, production ready, system to implement asynchronous tasks. They call it the uWSGI Spooler. Citing the uWSGI documentation:

The Spooler is a queue manager built into uWSGI that works like a printing/mail system.

You can enqueue massive sending of emails, image processing, video encoding, etc. and let the spooler do the hard work in background while your users get their requests served by normal workers.

A spooler works by defining a directory in which “spool files” will be written, every time the spooler find a file in its directory it will parse it and will run a specific function.

The uWSGI Spooler is very easy to configure!

Install uwsgidecorators package

To interact with uWSGI spooler from your Python code it is really convenient to install the uwsgidecorators package. Unfortunately there isn’t a recent version of uwsgidecorators on PyPI, but you can install it using apt:

sudo apt install python3-uwsgidecorators

If you use a virtualenv for your Django project (as you should) you can link the uwsgidecorator module in your virtualenv:

ln -s /usr/lib/python3/dist-packages/uwsgidecorators.py /path/to/your/virtualenv/lib/python3.6/site-packages

Create the directory for the “spool files”

First of all you have to create the directory where uWSGI will save the “spool files” used by the spooler:

mkdir -p /home/ubuntu/tasks
sudo chown www-data.www-data /home/ubuntu/tasks

Please make sure that the directory is writable by the user running uWSGI, that is www-data in the example.

Create a tasks module in your Django application

Assuming that you have a Django app named app1 you should create a module named tasks.py in the app directory. The content of the module should be something like this:

import logging
try:
    from uwsgidecorators import spool
except:
    def spool(func):
        def func_wrapper(**arguments):
            return func(arguments)
        return func_wrapper

import django
django.setup()

from .models import Model1

logger = logging.getLogger(__name__)

@spool
def long_running_task(arguments):
    id = arguments['id']
    obj1 = Model1.objects.get(pk=id)
    obj1.long_running_model_method()

The try/except construct will let you test the code even when not running in a uWSGI application server, for instance when running locally using ./manage.py runserver. In that case the task will be executed synchronously, as it was a regular python function call.

The long_running_task function is the task you will invoke using the uWSGI Spooler. As you can see you have to decorate it with the @spool decorator, and the parameters for the task are passed in a dictionary called arguments.

In this example I’m using the Django ORM to get an instance of a model and to call a model method that will take a long time to complete.

Call the asynchronous task from your Django view

Here you can find how to call the asynchronous task from an example view:

from django.shortcuts import redirect
from django.contrib import messages

from .tasks import long_running_task

def async_view(request, id):
    long_running_task(id=id)
    messages.add_message(request, messages.SUCCESS,
        'Task started correctly')
    return redirect('some_other_view')

Here I call the task, then I use the Django messages framework to show the user a confirmation message on the next page view, and finally I redirect the user to another view.

Configure uWSGI to use the spooler

To configure the uWSGI Spooler you should edit the uWSGI configuration file I presented in the post Django – NGINX: deploy your Django project on a production server. The file is located here: /etc/uwsgi/apps-enabled/django.ini. The content of the file should be something like this:

[uwsgi]
chdir = /home/ubuntu/django_project # customize with your django installation directory
env = DJANGO_SETTINGS_MODULE=project.settings.production # customize with your settings module
wsgi-file = project/wsgi.py # customize with the relative path to your wsgi.py file
workers = 1
spooler-chdir = /home/ubuntu/django_project # customize with your django installation directory
spooler = /home/ubuntu/tasks/
import = app1.tasks

Restart uWSGI with:

service uwsgi restart

You will find the uWSGI logs in /var/log/uwsgi/apps/django.log. Therefore you can check them to see if the Python process started correctly or there are issues.

In the log file you’ll also find messages about the Spooler, something like this:

Mon Nov 18 14:50:45 2019 - [spooler /home/ubuntu/tasks pid: 8646] managing request uwsgi_spoolfile_on_www.example.com_8647_3_1383635828_1574067630_419736 ...

Conclusion

In this post I’ve shown how to setup Django asynchronous tasks using uWSGI Spooler. That is a simple and convenient way to perform long running tasks outside the request-response cycle. Especially if you already use uWSGI application server.

I used this technique in many different projects, for example an “Instagram like” photo and video sharing platform I built from scratch for the popular online magazine mtb-mag.com. In that case videos are processed asynchronously to make two versions suitable for all devices, in standard and high definition.

Please let me know if you have comments or question by leaving a reply below. Happy coding!

Django NGINX: deploy your Django project on a production server

2019-11-13T09:46:56+00:00

Django NGINX is a popular and well tested combination used to deploy web applications in production. In this post I will explain the steps needed to have your Django project deployed on a production server using Ubuntu 18.04.

To have Django NGINX deployed on your production server follow these simple steps.

Table of Contents

Django Nginx: it's better together!
Install required packages using apt
Create directories for your static and media files
Setup your Django project and install requirements
Collect static files
Configure uWSGI to host your Django project
Configure NGINX to serve your application
Enjoy your Django NGINX application
Extra step: automate all these steps with Ansible!

Django Nginx: it's better together!

Django is a web framework written in Python. It lets you develop and prototype web applications with little effort. It comes with many built-in features that are often useful in a web application, such as: session management, user authentication, an object relational mapper, a template language.

NGINX is a very fast and configurable web server that is very well suited to serve static files and act as a reverse proxy for other applications. In this tutorial I'll explain how to configure NGINX to serve static files (images, javascript, css, ...) and to proxy requests to your Django application.

Install required packages using apt

sudo apt install nginx uwsgi uwsgi-plugin-python3

Why do you need uWSGI? In very simple terms NGINX on its own cannot run a Python process to host your application, for this you’ll need a so called application server that will host a Python process running your Django project. NGINX and uWSGI will “talk” each other using the uwsgi protocol.

Create directories for your static and media files

Static files are “not-python” files needed by your Django project, for example Javascript, CSS and images. Media files will be the files uploaded by the users of your application. Not every application will let users upload files, but it’s a very common scenario. Django will not serve static and media files by itself. We’ll leverage NGINX to serve them.

First of all you have to create the directories. Here I assume that you are currently using the user ubuntu with the default home directory /home/ubuntu:

mkdir -p /home/ubuntu/static /home/ubuntu/media
sudo chown www-data.www-data /home/ubuntu/media

The second command will make the user named www-data the owner of the /home/ubuntu/media directory. www-data will be the user running your Python process in uWSGI, and that user should be able to write in the media directory to correctly save user uploaded files.

Setup your Django project and install requirements

This step really depends on your particular Django application, for the purpose of this tutorial I will assume that your Django project is installed in the directory /home/ubuntu/django_project/ with the following structure:

/home/ubuntu/django_project/
├── app1
│   ├── admin.py
│   ├── __init__.py
│   ├── migrations
│   │   └── __init__.py
│   ├── models.py
│   ├── tests.py
│   ├── views.py
├── manage.py
└── project
   ├── __init__.py
   ├── settings
  │   └── __init__.py
  │   └── base.py
  │   └── production.py
   ├── urls.py
   ├── wsgi.py

Also I will assume that you installed all your Python requirements, for example using apt or pip.

I always follow a best practice when starting a new Django project, by splitting the monolithic settings.py file in different files, one for each deploy environment (local, test, production, …). You can read a more in depth explanation of this approach if you aren’t used to it.

In our case Django will use the module project/settings/production.py for his settings. Here we set the STATIC_ROOT and MEDIA_ROOT variables to the directories we created at step 2:

from .base import *

ALLOWED_HOSTS = [ 'www.example.com' ] # customize with your domain name

DATABASES = {
    'default': { ... } # here the configuration for your database
}

STATIC_ROOT = '/home/ubuntu/static'
MEDIA_ROOT = '/home/ubuntu/media'

Collect static files

Run the following command to collect all static files for your Django project:

./manage.py collectstatic

This command will copy all static files (Javascript, CSS, images) for all your Django apps in the STATIC_ROOT directory configured in production.py. For instance /home/ubuntu/static.

Configure uWSGI to host your Django project

Create a file named django.ini in the /etc/uwsgi/apps-enabled/ directory. The content of the file should be something like this:

[uwsgi]
chdir = /home/ubuntu/django_project # customize with your django installation directory
env = DJANGO_SETTINGS_MODULE=project.settings.production # customize with your settings module
wsgi-file = project/wsgi.py # customize with the relative path to your wsgi.py file
workers = 1

Restart uWSGI with:

service uwsgi restart

You should find the uWSGI logs in /var/log/uwsgi/apps/django.log. Therefore you can check them to see if the Python process started correctly or there are issues.

Configure NGINX to serve your application

Create a file named django in the /etc/nginx/sites-enabled/ directory. The content of the file should be something like this:

server {
    listen 80;
    server_name www.example.com; # customize with your domain name

    location / {
        # django running in uWSGI
        uwsgi_pass unix:///run/uwsgi/app/django/socket;
        include uwsgi_params;
        uwsgi_read_timeout 300s;
        client_max_body_size 32m;
    }

    location /static/ {
       # static files
       alias /home/ubuntu/static/; # ending slash is required
    }

    location /media/ {
        # media files, uploaded by users
        alias /home/ubuntu/media/; # ending slash is required
    }
}

Restart NGINX with:

service nginx restart

Enjoy your Django NGINX application

Point the browser to your domain, and you should see your Django application in all of its glory!

Extra step: automate all these steps with Ansible!

If you have to manage many different Django projects on many different servers, certainly you’ll find that automating stuff is always a good idea.

Read my post on How to deploy a Django project in 15 minutes with Ansible to automate all the steps described here.

In conclusion, I hope that this post helped you with configuring Django NGINX to deploy your Django project on a production server. Please let me know if you have questions leaving a comment in the area below or contacting me by email or social account.

A Django custom command to write model fields on an Excel file

2017-11-09T17:36:15+00:00

Suppose you need to write down your model fields on an Excel file, for example to complement the documentation of your code.

Django has built in functions to introspect models and fields of an app, and you can leverage this API to have the information you need.

You can use the following Django custom command:

# coding=utf-8 from __future__ import unicode_literals import logging from django.core.management.base import BaseCommand, CommandError from django.apps import apps from xlsxwriter.workbook import Workbook logger = logging.getLogger(__name__) class Command(BaseCommand): help = 'Export all models of a given app to Excel file' def add_arguments(self, parser): parser.add_argument('app_labels', nargs='+', type=str) parser.add_argument('output', type=str) parser.add_argument('--exclude-models', dest="exclude_models_file", type=file) parser.add_argument('--include-reverse-relations', action='store_true', dest="include_reverse_relations", default=False) def handle(self, *args, **options): workbook = Workbook(options['output'], {'constant_memory': True}) bold = workbook.add_format({'bold': True}) headers = ['Field', 'Type'] exclude_models = [] if options['exclude_models_file']: exclude_models = options['exclude_models_file'].read().splitlines() for app_label in options['app_labels']: app_config = apps.get_app_config(app_label) app_models = app_config.get_models(include_auto_created=False) for model in app_models: model_name = model.__name__ if model_name in exclude_models: continue worksheet_name = '%s - %s' % (app_label, model_name) if len(worksheet_name) > 31: worksheet_name = '%s - %s' % (app_label[:3], model_name) worksheet = workbook.add_worksheet(worksheet_name) worksheet.write_row(0, 0, headers, bold) worksheet.set_column(0, len(headers), 30) fields = model._meta.get_fields() row = 1 for f in fields: try: fieldname = f.verbose_name except: fieldname = f.name fieldtype = f.__class__.__name__ if (options['include_reverse_relations'] or fieldtype not in ('ManyToOneRel', 'ManyToManyRel', 'OneToOneRel')): worksheet.write(row, 0, fieldname.strip()) worksheet.write(row, 1, fieldtype) row += 1 workbook.close()

You can run the command like this:

./manage.py excel_models app1 app2 models.xlsx

And it will write all the models of app1 and app2 on the models.xlsx file. Every model will be written on a different sheet.

The option _–exclude-models_can be used to specify a text file with models to exclude from Excel file (one per line).

The option –include-reverse-relations (default false) can be used to include also reverse relations of your models, that are fields that are not part of the models itself, but are reverse fields from other models that have for instance a ForeignKey to the model.

The command depends on the XlsxWriter package to create the Excel file.

A simple Python email gateway

2017-05-10T13:23:11+00:00

Say you have a Django web application that you want to integrate with emails to make it possibile to send data and files to your web application over SMTP.

The good news is that Python has a simple SMTP daemon in the standard library, together with modules to parse emails. Let’s see how to create a simple email gateway on top of these tools.

Using the following Python module you can setup a simple pure Python email server on port 25, parse the incoming emails and make a multipart/form-data POST HTTP request to your web application. You’ll need the awesome Requests library for the HTTP request to work.

import smtpd import asyncore import logging import email from email.header import decode_header import requests logger = logging.getLogger(__name__) class CustomSMTPServer(smtpd.SMTPServer): def process_message(self, peer, mailfrom, rcpttos, data): try: logger.debug('Receiving message from: %s:%d' % peer) logger.debug('Message addressed from: %s' % mailfrom) logger.debug('Message addressed to: %s' % str(rcpttos)) msg = email.message_from_string(data) subject = '' for encoded_string, charset in decode_header(msg.get('Subject')): try: if charset is not None: subject += encoded_string.decode(charset) else: subject += encoded_string except: logger.exception('Error reading part of subject: %s charset %s' % (encoded_string, charset)) logger.debug('Subject: %s' % subject) text_parts = [] attachments = {} logger.debug('Headers: %s' % msg.items()) # YOU CAN DO SOME SECURITY CONTROLS HERE if (not mailfrom.endswith("@example.com") or not msg.get('Mail-Header') == 'expected value'): raise Exception("Email not trusted") # loop on the email parts for part in msg.walk(): if part.get_content_maintype() == 'multipart': continue c_type = part.get_content_type() c_disp = part.get('Content-Disposition') # text parts will be appended to text_parts if c_type == 'text/plain' and c_disp == None: text_parts.append(part.get_payload(decode=True).strip()) # ignore html part elif c_type == 'text/html': continue # attachments will be sent as files in the POST request else: filename = part.get_filename() filecontent = part.get_payload(decode=True) if filecontent is not None: if filename is None: filename = 'untitled' attachments['file%d' % len(attachments)] = (filename, filecontent) body = '\n'.join(text_parts) except: logger.exception('Error reading incoming email') else: # this data will be sent as POST data data = { 'subject': subject, 'body': body, 'from': mailfrom, } url = 'http://yourserver.example.com/email-receive/' logger.debug('Sending data and files to %s' % url) try: r = requests.post(url, data=data, files=attachments) r.raise_for_status() except: logger.exception('Error in server request') logger.debug(data) logger.debug(attachments) else: logger.debug('Response from server: %s %s' % (r.status_code, r.text)) return if __name__ == '__main__': ch = logging.StreamHandler() ch.setLevel(logging.DEBUG) formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') ch.setFormatter(formatter) logger.setLevel(logging.DEBUG) logger.addHandler(ch) server = CustomSMTPServer(('111.111.111.111', 25), None) # use your public IP here asyncore.loop()

In a production environment you can setup supervisor to launch your simple SMTP server. With a simple configuration like this:

[program:smtp_server]
command = /home/ubuntu/.virtualenvs/venv/bin/python /home/ubuntu/mail_gateway/smtp_server.py
user = root

Then try to send an email to your server, if the server public IP has been associated to a DNS record yourserver.example.com you can use something like test@yourserver.example.com as a recipient, and you should see the incoming email in the logs.

Have fun with emails!

How to deploy a Django project in 15 minutes with Ansible

2017-05-02T11:01:47+00:00

In this tutorial I will explain how to deploy a Django project in 15 minutes with Ansible. I will assume that you are a Django developer and you have built and tested a project locally. It’s time to deploy the project on a public server to let users access your awesome application.

If you are new in deploying Django on a production server you can read my post Django – NGINX: deploy your Django project on a production server to have a basic introduction on the steps needed.

So you need a VPS with an SSH access, then you will access the server, install and configure all necessary software (web server, application server, database server), create a database user, configure Django to use it, copy your Django project on the server, migrate the database, collect static files, trial and error, fix, trial and error, …

All this boring stuff will take some good hours that you should definitely spend in a more profitable way, don’t you think? The good news is that you can automate almost all the work needed to go from a vanilla VPS to a fully deployed server hosting your Django project.

Follow this tutorial and I’ll show you how to leverage the power of Ansible to automate all the needed steps in 15 minutes. Are you ready? Check the time on your clock and follow me!

Table of Contents

1. Setup the SSH access to your VPS
- Add the VPS address to your SSH configuration
- Configure SSH access without using a password
2. Deploy your Django project with Ansible
3. Update your Django project
4. Continuous delivery using Bitbucket Pipelines
5. Conclusions

1. Setup the SSH access to your VPS

There are plenty of good VPS providers out there, and the choice of the VPS is out of scope of this tutorial.

If you are looking for a cheap and powerful cloud provider to host your project I highly suggest DigitalOcean. It is very flexible and powerful, the price is very good for what they offer and you’ll get no surprises on your billing at the end of the month! Join by using this link and you’ll get a 100$ free credit to use in 60 days.

Here I assume that you already bought a Debian/Ubuntu based VPS with a public IP/hostname and root SSH access. I tested the procedure described in this tutorial using Ubuntu server 18.04.

Add the VPS address to your SSH configuration

If you use OpenSSH client on Linux/UNIX, you can add an entry like this in ~/.ssh/config:

Host yourserver
User root
Port 22
HostName yourserver.example.com

Pay attention to what you use on Host value, yourserver in this example, because this is the label you’ll use in the following steps to refer to this particular server.

In HostName you’ll configure the actual IP address or hostname of your VPS, according to the access data received by your provider.

Configure SSH access without using a password

To be able to connect to your VPS without using a password, you have to setup a public/private SSH key on your workstation (if you don’t already have one), this is very simple and can be done with the following command:

ssh-keygen

Then you can configure a password-less access to your VPS using the command:

ssh-copy-id yourserver

That’s it. You should now be able to connect to your VPS using SSH without entering your password every time.

2. Deploy your Django project with Ansible

Clone the template repository

I prepared a template repository of a Django project, you can clone it at the following address before proceeding:

git clone https://github.com/baxeico/django_ansible.git

In the repository you’ll find three directories:

django

This directory is a sample Django project created with the usual:

django-admin.py startproject yourproject

I renamed the root directory created by the command to be simply django.

It is just a bare template not meant to be used as is, but it should give you an idea on how the Django settings are supposed to be structured in this tutorial.

I always follow a best practice when starting a new Django project, by splitting the monolithic settings.py file in different files, one for each deploy environment (local, test, production, …). You can read a more in depth explanation of this approach if you aren’t used to it.

So inside yourproject sub-directory you’ll find a settings package containing different modules for different deploy environments:

base.py – This is the common settings which will be inherited (and possibly overridden) from all deploy environments;
local.py – This will be the DJANGO_SETTINGS_MODULE used in your local development environment. Nothing special here, it only makes Django to use sqlite as DB for local development;
production.py – This will be the DJANGO_SETTINGS_MODULE used in production. Here you’ll find that the database used by Django is defined using dj-database-url package, by means of an environment variable. Also the STATIC_ROOT settings variable is defined by means of an environment variable.

requirements

This directory contains the minimum Python requirements of your project, you’ll find three files there:

base.txt – contains the basic requirements for all deploy environment (only Django in the example, it’s a Django project after all, isn’t it?);
local.txt – it inherits from base.txt and adds Ansible as a requirement for your local development environment. Version 2.3.0 is the latest Ansible version at the time of writing;
production.txt – it also inherits from base.txt and adds dj-database-url as a requirement for your production environment. It will be used to configure the database used by Django by means of the environment variable DATABASE_URL.

Notice how this structure mimics the different Django settings modules described above. You can install the local requirements using this command:

pip install -r requirements/local.txt

I strongly suggest to use Virtualenv, together with virtualenvwrapper to create and manage an isolated environment for your project.

ansible

In this directory you’ll find the most interesting part, that is a set of Ansible playbooks to automate the installation and configuration of the server and the deployment of your project.

I suggest to copy the ansible directory in your project root, and eventually adapt the playbooks to your needs.

The first thing to customize is the hosts file. This is the file where you could list all the hosts controlled by Ansible. In this example you should have only one entry, corresponding to the Host value you entered in SSH client configuration.

Then you can proceed to rename the file host_vars/yourserver, using the label you gave to your server in the hosts file. Here you’ll also find some variables used in the playbooks.

Here is a brief description of each playbook you’ll find:

config_files.yaml – Copy nginx and uwsgi configuration files on remote server.
deploy.yaml – Deploy your Django project on the server, pulling the master branch from your GIT repository, installing all needed production requirements, running migrate and collectstatic and restarting uwsgi.
packages.yaml – Install needed software packages on remote server using apt.
postgresql.yaml – Create and configure the access to a Postgresql database used by your Django project on remote server. Notice how the database password is randomly generated and stored in the local file postgresqlpasswd. The password will be automatically inserted in the DATABASE_URL environment variable, to be used in Django production settings.
system.yaml – Create an user ubuntu on remote server, together with a private/public SSH key pair. The public key is returned as output when you run the playbook, to be used as a “deploy key” on the server. More details later on this step.
upgrade.yaml – Upgrade all apt packages on remote server.

Push your Django project in a publicly accessible GIT repository

Probably you already have your project sources hosted on a public GIT repository like Github or Bitbucket. If you don’t, this is a perfect time to do it! I personally use Bitbucket, because it offers an unlimited number of private repositories.

In the following steps I assume that you have your Django project hosted on Bitbucket, under this path:

ssh://git@bitbucket.org/youruser/yourproject.git

Let the Ansible automation begins!

Inside the ansible directory, run the following command:

./ansible.sh system.yaml

You should see an output similar to this:

ok: [yourserver] => {
    "changed": false,
    "msg": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1kvkW9... ansible-generated on yourserver.example.com\n"
}

What you find in the msg variable is the public SSH key generated for the ubuntu user on remote server. You should copy the public key and add it as a “deploy key” in the settings of your GIT repository.

A deploy key is a read-only SSH key that will be used to clone your repository from the remote server. You can find more details in the Bitbucket and Github documentation.

Now you are ready to complete the deploy of your Django project! Run the following commands inside the ansible directory:

./ansible.sh packages.yaml
./ansible.sh postgresql.yaml
./ansible.sh config_files.yaml
./ansible.sh deploy.yaml

If all goes well you should be able to reach your Django project on your remote server public address, on port 80.

3. Update your Django project

Keeping your Django project updated on the remote server is very easy in this setup. You only need to push your changes to your GIT repository (on the master branch) and then you can run the following command inside the ansible directory:

./ansible.sh deploy.yaml

This playbook will perform the following tasks for you:

pull the updated master branch from your GIT repository;
eventually install new Python production requirements, and update the existing ones;
migrate the database to the latest version;
update your static files with collectstatic;
restart uwsgi.

4. Continuous delivery using Bitbucket Pipelines

In the follow up article Bitbucket Pipelines and Ansible: Continuous delivery for your Django project I explained how you can use Ansible together with Bitbucket Pipelines to implement a simple yet powerful continuous delivery setup for your Django project.

5. Conclusions

In this article I explained how to deploy a Django project in 15 minutes with Ansible. The key here is in following some Django best practices and leveraging the power on Ansible to quickly deploy a Django project on a remote server. I think that the time spent to learn and master those technologies is worth the effort for the time you’ll save in the future and all the human errors you’ll avoid by automating and standardizing your processes.

Of course this is only the tip of the iceberg of what you could do and automate with Ansible, but I found this simple approach to be already a big time saver in my daily work.

Have fun with Ansible!

How to add CORS headers to a Django view for use with a Cordova app

2016-05-27T16:42:17+00:00

Suppose you want to access some JSON data from a mobile app using Cordova. You have to bypass CORS restrictions in the web view, and to do that you have to provide some HTTP headers in your Django views.

You can do this pretty easily by using this mixin in your Class Based Views:

from django.http import HttpResponse

class AllowCORSMixin(object):

    def add_access_control_headers(self, response):
        response["Access-Control-Allow-Origin"] = "*"
        response["Access-Control-Allow-Methods"] = "GET, OPTIONS"
        response["Access-Control-Max-Age"] = "1000"
        response["Access-Control-Allow-Headers"] = "X-Requested-With, Content-Type"

    def options(self, request, *args, **kwargs):
        response = HttpResponse()
        self.add_access_control_headers(response)
        return response

Use it like in the example below. Here used together with Django Braces app:

from django.views.generic import View

from braces.views import AjaxResponseMixin, JSONResponseMixin

class JsonView(JSONResponseMixin, AjaxResponseMixin, AllowCORSMixin, View):

    def get_ajax(self, request, *args, **kwargs):

        # ... get some data ...

        response = self.render_json_response({ 'data': data })
        self.add_access_control_headers(response)
        return response

How to migrate your existing Django project to Heroku

2016-05-18T14:15:45+00:00

Recently I had some fun with Heroku, the well known PaaS provider. I had a small personal Django project I use for invoicing that I ran locally with ./manage.py runserver when needed. That was a perfect candidate for the Heroku free plan because I need to access the app only occasionally.

In this tutorial I assume you have a basic knowledge of what Heroku is, and that you already know how to create and deploy a Python project on Heroku. In case you miss some basic information you can refer to the good Getting started tutorial on Heroku with Python.

Table of Contents

How to structure your Django project for Heroku
How to configure your Django project for Heroku
Create an Heroku application for your Django project
Migrating data
Media files on AWS S3

How to structure your Django project for Heroku

Here I focus on my use case, which was to migrate an existing Django project on Heroku platform. My existing Django project was structured in accordance to the best practices I read in the wonderful Two Scoops of Django book, so my project structure was similar to this:

django/
├── project
│   ├── __init__.py
│   ├── settings
│   │   ├── __init__.py
│   │   ├── base.py
│   │   ├── local.py
│   │   └── production.py
│   ├── urls.py
│   ├── wsgi.py
├── app
│   ├── __init__.py
│   ├── admin.py
│   ├── models.py
│   ├── tests.py
│   ├── urls.py
│   ├── views.py
└── manage.py
requirements
├── base.txt
├── local.txt
└── production.txt

See how I have different settings for each deploy environment: one for local development, one for production. All common settings go in base.py.

See also how I have a different requirements file for each environment, for example I have django-debug-toolbar in local.txt, while I have psycopg2 in production.txt. All common requirements are in base.txt.

For Heroku I created a file heroku.txt under requirements/ directory with the following content:

-r base.txt

dj-database-url==0.3.0
psycopg2==2.6.1
whitenoise==2.0.2
django-storages==1.4.1
boto==2.38.0

How to configure your Django project for Heroku

On Heroku I will use Postgres as database backend (which is free , and dj-database-url to configure the database parameters using an environment variable, which is already set by Heroku.

I’ll also use whitenoise to handle static files directly from django and django-storages to handle media files on Amazon S3.

The first thing you have to do to tell Heroku you are deploying a python project is to put a requirements.txt file in the root of your git project. You can create a file with this content to include the requirements specific to Heroku:

-r requirements/heroku.txt

Then you have to do some modification to your settings, for deploy on Heroku platform:

# Parse database configuration from $DATABASE_URL
import dj_database_url
from os import environ

from .base import *

ALLOWED_HOSTS = ['yourappname.herokuapp.com']

DATABASES = {
 'default': dj_database_url.config()
}

WSGI_APPLICATION = 'yourproject.wsgi_heroku.application'
STATICFILES_STORAGE = 'whitenoise.django.GzipManifestStaticFilesStorage'
STATIC_ROOT = root('static_root')

INSTALLED_APPS += ('storages',)
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = environ['AWS_ACCESS_KEY_ID']
AWS_SECRET_ACCESS_KEY = environ['AWS_SECRET_ACCESS_KEY']
AWS_STORAGE_BUCKET_NAME = 'yourbucketname'

Then you have to edit your wsgi.py file to make use of whitenoise for static files, the file whould look something like this. Save it in a different wsgi_heroku.py file:

import os
from django.core.wsgi import get_wsgi_application
from whitenoise.django import DjangoWhiteNoise

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "yourproject.settings.heroku")

application = get_wsgi_application()
application = DjangoWhiteNoise(application)

Create an Heroku application for your Django project

Now you can create you Heroku application:

heroku create --region eu yourappname

In the root directory of your project you should have a Procfile, used by Heroku to know how to launch the web worker, in our case the Procfile should contain just one line like the following:

web: gunicorn --pythonpath django yourproject.wsgi_heroku --log-file -

Now push the code on Heroku:

git push heroku master

Then you can create the database schema with the usual migrate command, run on the Heroku instance:

heroku run python django/manage.py migrate

Migrating data

If you have some data to migrate you can use a couple of command from core Django

./manage.py dumpdata yourapp > yourapp/fixtures/app_data.json

Then you can import the data on Heroku instance:

heroku run python django/manage.py loaddata app_data

Media files on AWS S3

If your application needs some user uploaded data, you can use Amazon S3 to store the files, by using django-storages app to configure the correct Storage for you media files.

The configuration of django-storages is out of scope for this tutorial, but you can find some good tutorials on the web.

A simple script to update a DB migration in Django 1.7+

2015-10-22T10:41:37+00:00

During the development of a Django model on your local machine is it often necessary to refine the most recent migration to cope with updates to the model, without polluting the migrations of the app with a new migration for each local update.

South had the update flag for the schemamigration command, but I didn’t find a similar functionality in the Django builtin makemigrations command introduced in Django 1.7.

So I put togheter a simple bash script to automate the process.

#!/bin/bash if [ -z $1 ]; then echo "Usage: $0 app_label" exit 1 fi app=$1 # this is the name of the app where we want to update the last migration # get the list of known migrations for the app migrations=(`./manage.py showmigrations $app | awk '{print $2}' | tail -2`) if [ ${#migrations[@]} == 1 ]; then # there is just one migration in the list # here we are updating the initial migration previous_migration=zero current_migration=${migrations[0]} # should be 0001_initial else # there is more than one migration in the list # get the previous one to go back to # and the current one to update previous_migration=${migrations[0]} current_migration=${migrations[1]} fi # go back to the previous migration ./manage.py migrate $app $previous_migration # remove the current, outdated migration rm $app/migrations/${current_migration}.* # create a new migration ./manage.py makemigrations $app # migrate the DB to the new migration ./manage.py migrate $app

Basically the script unapplies the last migration for a given app, update the migration and reapplies it. I find it a little time saver and it feels good to know that the script is doing the right things in the right order.

How to add a group choice combo box in a Django user profile form

2014-09-10T11:38:30+00:00

Assume that you have a Django project where each user belongs to just one group, say Registered or Admin, but not both.

You want to show a form in your front-end to let Admin users edit the user profiles, where each user profile is made with First name, Last name, Email and the user group.

This task can be accomplished very easily! What you need is a customized ModelForm to add the possibility to edit the user group together with the other fields, and a customized UpdateView to let you set the form initial data for the group field, and to save the changes correctly.

Here is the ModelForm:

from django import forms from django.contrib.auth.models import User, Group class UserProfileForm(forms.ModelForm): group = forms.ModelChoiceField(queryset=Group.objects.all(), required=True) class Meta: model = User fields = ['first_name', 'last_name', 'email', 'group']

That is a standard ModelForm for the User model, we have just added a group field that is a ModelChoiceField, that is a combo box with all available groups in our project as choices, if you want you can filter the queryset according to your needs. We have also restricted the fields shown in the user profile form to First name, Last name, Email and Group.

from django.contrib.auth.models import User from django.views.generic import UpdateView from .forms import UserProfileForm class UserProfileUpdateView(UpdateView): model = User def get_initial(self): initial = super(UserProfileUpdateView, self).get_initial() try: current_group = self.object.groups.get() except: # exception can occur if the edited user has no groups # or has more than one group pass else: initial['group'] = current_group.pk return initial def get_form_class(self): return UserProfileForm def form_valid(self, form): self.object.groups.clear() self.object.groups.add(form.cleaned_data['group']) return super(UserProfileUpdateView, self).form_valid(form)

This is the view we’ll use to update the user profile. We have overridden some methods of UpdateView:

get_initial is used to set the initial value for the custom group field with the current group the user belongs to. Note that when the user profile is edited for the first time the user will not belong to any group, but this is taken into account and no initial value will be provided to the field;
get_form_class is used to set the class used by the UpdateView to show and validate the form. We have set it to the one we created before;
form_valid is called by the UpdateView when the form submitted is valid. Here we remove the old group the user belonged to, and we add the new group selected in the form to the user. Remember that the groups field of the User model is a ManyToManyField, so we are simply using the Django ORM here.

That’s it, I hope that this how-to helps you with your Django project! Please let me know if you found my approach useful in the comments.

PUT and DELETE HTTP requests with Django and jQuery

2014-06-25T11:42:13+00:00

Sometimes it could be useful and elegant to have a Django view responding to more that GET and POST requests, implementing a simple REST interface. In this post I’ll show how you can support PUT and DELETE HTTP requests in Django.

Table of Contents

Your goal
The problems
The solution
What about the function based views?
Conclusion

Your goal

To put it in examples your goal is to have a View like this:

from django.views.generic import View
from django.shortcuts import redirect, render
from django.forms.models import modelform_factory
from django.http import HttpResponse

from .models import MyModel

class RestView(View):
  def get(self, request):
    # retrieve some object and render in template
    object = MyModel.objects.get(pk=request.GET['pk'])
    return render(request, 'object_detail.html',
                  {'object': object})

  def put(self, request):
    # create an object and redirect to detail page
    modelform = modelform_factory(MyModel)
    form = modelform(request.PUT)
    if form.is_valid():
        form.save()
    return redirect('restview')

  def delete(self, request):
    # delete an object and send a confirmation response
    MyModel.objects.get(pk=request.DELETE['pk']).delete()
    return HttpResponse()

And accessing the view by jQuery Ajax methods like this:

$.ajax({
    type: 'DELETE',
    url: '/restview',
    data: { pk: pk },
    success: function() {
        alert('Object deleted!')
    }
});

The problems

Unfortunately there are two problems that restrict you to implement the view and the javascript like described above:

Not all browsers support HTTP requests different from GET and POST. In some browsers the type: ‘DELETE’ in your ajax method will not work;
Django does not put data sent with a PUT or DELETE request in a request.PUT or request.DELETE dictionary, like it does with GET or POST data. This is for a good reason.

The solution

What you can do to solve these problems? You can use the so called POST TUNNELLING method, that is using a POST request, putting the HTTP header X_METHODOVERRIDE in the request.

In jQuery you can accomplish this by modifyng the ajax method a little bit:

$.ajax({
    type: 'POST',
    url: '/restview',
    data: { pk: pk },
    success: function() {
        alert('Object deleted!')
    },
    headers: { 'X_METHODOVERRIDE': 'DELETE' }
});

So now you have a “standard” type: ‘POST’ and you’ve added the X_METHODOVERRIDE=DELETE HTTP header to the request.

To support this in the Django side you can write a middleware, like this:

from django.http import QueryDict

class HttpPostTunnelingMiddleware(object):
    def process_request(self, request):
        if request.META.has_key('HTTP_X_METHODOVERRIDE'):
            http_method = request.META['HTTP_X_METHODOVERRIDE']
            if http_method.lower() == 'put':
                request.method = 'PUT'
                request.META['REQUEST_METHOD'] = 'PUT'
                request.PUT = QueryDict(request.body)
            if http_method.lower() == 'delete':
                request.method = 'DELETE'
                request.META['REQUEST_METHOD'] = 'DELETE'
                request.DELETE = QueryDict(request.body)
        return None

The middleware intercepts the HTTP_X_METHODOVERRIDE header, and act accordingly by forcing the HTTP method in the Django side and creating the request.PUT and request.DELETE QueryDict.

Assuming that you saved the above middleware in a file called middleware.py in your myapp Django app, you can add the middleware to your settings.py like this:

MIDDLEWARE_CLASSES = (
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'myapp.middleware.HttpPostTunnelingMiddleware',
)

Now your coding dream will come true, enjoy your simple REST View in Django, consuming it with jQuery!

What about the function based views?

You are not limited to a class-based view, but you can also use a function based view, by writing a view function similar to this:

def rest_view(request):
  if request.method == 'GET':
    # retrieve some object and render in template
    object = MyModel.objects.get(pk=request.GET['pk'])
    return render(request, 'object_detail.html',
                  {'object': object})

  elif request.method == 'PUT':
    # create an object and redirect to detail page
    modelform = modelform_factory(MyModel)
    form = modelform(request.PUT)
    if form.is_valid():
        form.save()
    return redirect('restview')

  elif request.method == 'DELETE':
    # delete an object and send a confirmation response
    MyModel.objects.get(pk=request.DELETE['pk']).delete()
    return HttpResponse('ok')

Conclusion

I hope that this post helped you to implement put and delete requests in Django. If you want you can read other interesting Django related posts here on my blog.

Django on windows: how to make Django run in CherryPy as a Windows service

2013-10-13T20:23:11+00:00

Recently a client asked me to investigate the possibility to have a Django application hosted in a Windows environment. I looked around a bit and there are many ways to achieve this goal. I found the “pure Python” approach described here simpler to deploy and configure, yet very powerful and production ready for my needs.

I will explain how to install Python, CherryPy and Django on Windows, and how to configure them to serve a Django application from the CherryPy WSGI server, running as a Windows service.

Table of Contents

Install Python 2.7
Install CherryPy
CherryPy server started as a Windows Service
Install Django
Give a powerful DB engine to your Django app: install MySQL!
Make your Django project an installable python package
Django running in CherryPy as a Windows service
Conclusions

Install Python 2.7

First of all you have to install the Python programming language support for Windows. I used Python 2.7.5 Windows Installer.

Then to be able to invoke the python command from the command line – no matter what the current work directory is – add Python to system path. Go to

Control panel > System > Advanced > Environment variables

look for path variable, click on Edit and add this at the end:

C:\Python27\;C:\Python27\Scripts;

You have to close the Command prompt and then reopen it to have the path variable updated.

Install CherryPy

CherryPy is a popular WSGI framework written in Python. Sometimes it is used instead of Django to develop full fledged web applications. Here we will use the powerful WSGI and HTTP server implementation in CherryPy to host our Django application. Install latest version of CherryPy using a convenient Windows installer.

CherryPy will be installed in the standard path C:\Python27\Lib\site-packages. To check your installation you can try the first tutorial by writing the following commands in the Command Prompt:

cd C:\Python27\Lib\site-packages\cherrypy\tutorial
python tut01_helloworld.py

Then open a web browser and go to http://127.0.0.1:8080 , if you see Hello World your CherryPy installation was successful.

CherryPy server started as a Windows Service

It is very convenient to have your CherryPy based WSGI server launched as a Windows Service. In this way you can control your WSGI server using the standard management tools of Windows. You have to install the pywin32 package to have Windows Service integration in Python.

You can find a simple example script in the CherryPy wiki for launching a CherryPy based app as a Windows Service:

Save the script with the name cherrypy_service.py, then you can use the following commands to install the service in Windows and control the service afterwards:

Install service with:

python cherrypy_service.py install

and start it with

python cherrypy_service.py start

If you do modifications to the script, use

python cherrypy_service.py update

to update the service, then

python cherrypy_service.py restart

to restart the modified service. You can also configure the service to start automatically from:

Control panel > Administration tools > Services

Install Django

Django is a popular web framework written in Python. A complete description of Django and how to use it is beyond the scope of this how-to, but I suggest you to look at the wonderful tutorials in the Django documentation.

Download the latest version of Django, extract the tar.gz using WinZip (unfortunately tar.gz is not supported in my version of Windows), then install Django with the usual:

python setup.py install

Try your Django installation with the following command in the Command Prompt

python -c "import django; print(django.get_version())"

if Django is installed correctly it will print the version number of Django, for instance 1.5.4. Now you can create your Django project and apps as described in the first tutorial in the Django documentation.

Give a powerful DB engine to your Django app: install MySQL!

Download the MySQL 5.5 windows installer and launch it; the installer will ask you some configuration preferences, the most important is the root password for the DB administration

Enter mysql shell with your MySQL root account (assuming you chose mysecret as your root password):

mysql -uroot -pmysecret

In the mysql shell create a test database and a test user with all privileges for the test DB:

CREATE DATABASE test_django;<br /> CREATE USER django@localhost IDENTIFIED BY 'django';<br /> GRANT ALL PRIVILEGES ON test_django.* TO django@localhost;

To be able to use MySQL as a DB backend in Django you have to install a last package: MySQL-python, by downloading and launching a convenient Windows installer.

Then you can configure Django to use the newly created DB, by modifyng the DATABASES setting in the settings.py module of your Django project:

DATABASES = {
  'default': {
    'ENGINE': 'django.db.backends.mysql',
    'NAME': 'test_django',
    'USER': 'django',
    'PASSWORD': 'django',
    'HOST': '', # Empty for localhost through domain sockets
    'PORT': '', # Set to empty string for default.
  }
}

Now you can try syncdb to check the correct installation and configuration of Django and MySQL:

python manage.py syncdb

if all goes well it will create some “internal” Django tables in the MySQL database and it will ask you to create a Superuser account for your Django project.

Make your Django project an installable python package

That’s simple! Just download ez_setup.py save it in the Django project folder (where manage.py resides). Then in the same folder create a new file setup.py with this content:

from ez_setup import use_setuptools
use_setuptools()
from setuptools import setup, find_packages
setup(
  name = "django_site",
  version = "0.1",
  packages = find_packages(),
  author = "John Doe",
  author_email = "john@example.com",
  description = "Test Django",
  include_package_data = True
)

Now you can install your Django project by doing:

python setup.py install

Django running in CherryPy as a Windows service

This great article explains how to serve a Django application from the CherryPy WSGI server, I borrowed the code given in their Bitbucket repo and I modified it a little to better fit my needs. I also added a script to make our CherryPy server launched as a Windows service, as described before. You can download a zip package from Google Drive with all the code.

You can use the following command to install the Windows service:

python cherrypy_django_service.py install

Then you can start the service with:

python cherrypy_django_service.py start

The script cherrypy_django_service.py assumes your Django project is installed in a standard Python path; this is guaranteed if you installed your Django project as a Python package as described before.

Also, the script assumes that your Django project is called django_site. If your Django project has a different name (likely) you have to edit cherrypy_django_service.py accordingly.

If everything works you should see your Django site at the address:

http://127.0.0.1:8090

You can change the port in the cherrypy_django_service.py script, together with other CherryPy options, which are however beyond the scope of this how-to.

Conclusions

The method described here is IMHO a quite simple and straightforward way to deploy Django on Windows. I’m particularly interested in your feedback about this approach and the results you obtained. Also the scalability and the performance of this solution is a very interesting discussion topic for me. Please leave a comment below if you have something to share.