Guguweb - ansible

Bitbucket Pipelines and Ansible: Continuous delivery for your Django project

2020-01-03T08:33:41+00:00

“Bitbucket Pipelines and Ansible: Continuous delivery for your Django project” was originally published as a guest post on Bitbucket Blog.

Table of Contents

Introduction
Let the automation begin!
Setup your pipelines
Conclusion

Introduction

Ansible is a powerful tool for the automation of your server administration. It can be used both to install and configure the software needed for your application and to deploy a new version of your application.

Bitbucket Pipelines are basically Docker containers, hosted in the Bitbucket infrastructure, that you can launch automatically to build or deploy your code. They can be attached to “events” happening in your repository, usually a push on one branch.

These tools are complementary to each other, because on Ansible you can define exactly what you want to do on your target server to deploy your application, while on Bitbucket Pipelines you can configure and launch a container to perform the actual deploy.

In a previous post I described how to deploy a Django project using Ansible. Here I explain how to make a step further and deploy a Django project using Bitbucket Pipelines and Ansible.

The use-cases for deploying a Django project automatically are many. Even if your Django project has a simple architecture, there are many steps you have to perform in the right order to complete a successful deploy in the general case. As a bare minimum you’ll have to: update the application code, install or update project dependencies, migrate the database, collect static files, restart the application server.

Automating all these steps in a reproducible and formal way will save you a lot of time in the long run and will help you to avoid human errors. Think about how many times you’ll deploy a new version of your application in the future.

If your architecture is more complex, the number of steps to perform will be larger and the deploy process will be more error-prone, so the benefits of an automated deploy will be even greater.

Let the automation begin!

You can clone or fork this Bitbucket repository to follow along with this tutorial.

First thing first, add a file named bitbucket-pipelines.yml to the root of your repository. The content of the file should be similar to this:

# use the official Python 2.7.16 docker image
image: python:2.7.16

pipelines:
  branches:
    # deploy only when pushing to the master branch
    master:
      - step:
          name: Deploy to prod
          # this is the name of the Bitbucket Deployment
          deployment: Production
          caches:
            # cache the Ansible installation
            - pip
          script:
            # install Ansible
            - pip install ansible==2.8.0
            # go into the ansible directory
            # in this same repository
            - cd ansible
            # perform the actual deploy
            - ansible-playbook -i ./hosts deploy.yaml

Let’s examine the configuration line by line:

On line 2 you define the docker image you want to use;
On line 7 you define the branch that will trigger the deploy (master in this example);
On line 11 you define the deployment, that is basically an environment (managed by Bitbucket) where you can define some variables. In this way you can have different deployments (staging, production, …) with different variables that you can use in your build/deploy steps. In this example I do not use this feature, and I merely use one of the default Bitbucket deployments (named Production) without configuring any variable in this environment;
On line 14 you are asking Bitbucket to cache the installation of packages installed by pip. This is useful to avoid to reinstall every time the Python libraries you need for your deployment (Ansible 2.8.0 in this example) from scratch. The build time for Bitbucket Pipelines is limited to 50 minutes/month in the free plan, so it’s important to have the libraries already installed to save some build/deploy time.
From line 16 to line 22 you define the actual steps for deployment: install Ansible (if not already cached), go into the ansible directory, make the deploy. Please check the playbook ansible/deploy.yaml to see what it will actually do on the target server.

Ansible will perform the actual deploy by:

pulling the master branch from your Git repository;
installing all needed production requirements;
running Django migrate and collectstatic commands;
restarting the application server uwsgi.

Please check the Ansible playbook ansible/deploy.yaml to see what it will actually do on the target server in details.

Commit and push the bitbucket-pipelines.yml file on the root of your Bitbucket repository.

Setup your pipelines

Now go on your repository settings and click Settings under the section PIPELINES:

Pipelines settings

Now click on Enable Pipelines:

Pipelines settings

If you have read my previous post on how to deploy a Django project using Ansible, you should know that Ansible work through an SSH connection made between the client host (the Bitbucket Pipeline docker container in this case) and the server where you want to deploy your code.

For this SSH connection to work you have to setup an SSH public/private keypair on the Bitbucket Pipeline, and allow the public key on your server authorized_keys.

Go on the settings of your repository, under PIPELINES navigate to SSH keys:

Pipelines settings

Now click on “Generate keys” to let Bitbucket generate a new SSH keypair for you.

Pipelines settings

Now you can copy the generated public key and paste it in the .ssh/authorized_keys file of your deploy target.

Pipelines settings

You’ll also have to add your server fingerprint to the Known hosts section in the same page. Insert your target server hostname in the “Host address” field, press the “Fetch” button and finally the “Add host” button.

Known hosts settings

You can refer to the Bitbucket documentation about SSH keys, if you need more details.

You can test that the deploy is working simply by pushing some changes to the master branch of the repository

If you want to push some changes on master, but you do not want to deploy the change automatically on the server, you can put the string [skip-ci] in your commit message.

Conclusion

In this tutorial I explained a simple yet powerful setup for the continuous delivery of a Django project. This is only a little example of what you can do with Bitbucket Pipelines and Ansible, but I hope that this helped you to understand the great potential of these technologies.

I think that learning and implementing this kind of automation in your projects is always worth the effort. Time saved and less human errors in the deploy process will make a big difference in the long run.

How to deploy a Django project in 15 minutes with Ansible

2017-05-02T11:01:47+00:00

In this tutorial I will explain how to deploy a Django project in 15 minutes with Ansible. I will assume that you are a Django developer and you have built and tested a project locally. It’s time to deploy the project on a public server to let users access your awesome application.

If you are new in deploying Django on a production server you can read my post Django – NGINX: deploy your Django project on a production server to have a basic introduction on the steps needed.

So you need a VPS with an SSH access, then you will access the server, install and configure all necessary software (web server, application server, database server), create a database user, configure Django to use it, copy your Django project on the server, migrate the database, collect static files, trial and error, fix, trial and error, …

All this boring stuff will take some good hours that you should definitely spend in a more profitable way, don’t you think? The good news is that you can automate almost all the work needed to go from a vanilla VPS to a fully deployed server hosting your Django project.

Follow this tutorial and I’ll show you how to leverage the power of Ansible to automate all the needed steps in 15 minutes. Are you ready? Check the time on your clock and follow me!

Table of Contents

1. Setup the SSH access to your VPS
- Add the VPS address to your SSH configuration
- Configure SSH access without using a password
2. Deploy your Django project with Ansible
3. Update your Django project
4. Continuous delivery using Bitbucket Pipelines
5. Conclusions

1. Setup the SSH access to your VPS

There are plenty of good VPS providers out there, and the choice of the VPS is out of scope of this tutorial.

If you are looking for a cheap and powerful cloud provider to host your project I highly suggest DigitalOcean. It is very flexible and powerful, the price is very good for what they offer and you’ll get no surprises on your billing at the end of the month! Join by using this link and you’ll get a 100$ free credit to use in 60 days.

Here I assume that you already bought a Debian/Ubuntu based VPS with a public IP/hostname and root SSH access. I tested the procedure described in this tutorial using Ubuntu server 18.04.

Add the VPS address to your SSH configuration

If you use OpenSSH client on Linux/UNIX, you can add an entry like this in ~/.ssh/config:

Host yourserver
User root
Port 22
HostName yourserver.example.com

Pay attention to what you use on Host value, yourserver in this example, because this is the label you’ll use in the following steps to refer to this particular server.

In HostName you’ll configure the actual IP address or hostname of your VPS, according to the access data received by your provider.

Configure SSH access without using a password

To be able to connect to your VPS without using a password, you have to setup a public/private SSH key on your workstation (if you don’t already have one), this is very simple and can be done with the following command:

ssh-keygen

Then you can configure a password-less access to your VPS using the command:

ssh-copy-id yourserver

That’s it. You should now be able to connect to your VPS using SSH without entering your password every time.

2. Deploy your Django project with Ansible

Clone the template repository

I prepared a template repository of a Django project, you can clone it at the following address before proceeding:

git clone https://github.com/baxeico/django_ansible.git

In the repository you’ll find three directories:

django

This directory is a sample Django project created with the usual:

django-admin.py startproject yourproject

I renamed the root directory created by the command to be simply django.

It is just a bare template not meant to be used as is, but it should give you an idea on how the Django settings are supposed to be structured in this tutorial.

I always follow a best practice when starting a new Django project, by splitting the monolithic settings.py file in different files, one for each deploy environment (local, test, production, …). You can read a more in depth explanation of this approach if you aren’t used to it.

So inside yourproject sub-directory you’ll find a settings package containing different modules for different deploy environments:

base.py – This is the common settings which will be inherited (and possibly overridden) from all deploy environments;
local.py – This will be the DJANGO_SETTINGS_MODULE used in your local development environment. Nothing special here, it only makes Django to use sqlite as DB for local development;
production.py – This will be the DJANGO_SETTINGS_MODULE used in production. Here you’ll find that the database used by Django is defined using dj-database-url package, by means of an environment variable. Also the STATIC_ROOT settings variable is defined by means of an environment variable.

requirements

This directory contains the minimum Python requirements of your project, you’ll find three files there:

base.txt – contains the basic requirements for all deploy environment (only Django in the example, it’s a Django project after all, isn’t it?);
local.txt – it inherits from base.txt and adds Ansible as a requirement for your local development environment. Version 2.3.0 is the latest Ansible version at the time of writing;
production.txt – it also inherits from base.txt and adds dj-database-url as a requirement for your production environment. It will be used to configure the database used by Django by means of the environment variable DATABASE_URL.

Notice how this structure mimics the different Django settings modules described above. You can install the local requirements using this command:

pip install -r requirements/local.txt

I strongly suggest to use Virtualenv, together with virtualenvwrapper to create and manage an isolated environment for your project.

ansible

In this directory you’ll find the most interesting part, that is a set of Ansible playbooks to automate the installation and configuration of the server and the deployment of your project.

I suggest to copy the ansible directory in your project root, and eventually adapt the playbooks to your needs.

The first thing to customize is the hosts file. This is the file where you could list all the hosts controlled by Ansible. In this example you should have only one entry, corresponding to the Host value you entered in SSH client configuration.

Then you can proceed to rename the file host_vars/yourserver, using the label you gave to your server in the hosts file. Here you’ll also find some variables used in the playbooks.

Here is a brief description of each playbook you’ll find:

config_files.yaml – Copy nginx and uwsgi configuration files on remote server.
deploy.yaml – Deploy your Django project on the server, pulling the master branch from your GIT repository, installing all needed production requirements, running migrate and collectstatic and restarting uwsgi.
packages.yaml – Install needed software packages on remote server using apt.
postgresql.yaml – Create and configure the access to a Postgresql database used by your Django project on remote server. Notice how the database password is randomly generated and stored in the local file postgresqlpasswd. The password will be automatically inserted in the DATABASE_URL environment variable, to be used in Django production settings.
system.yaml – Create an user ubuntu on remote server, together with a private/public SSH key pair. The public key is returned as output when you run the playbook, to be used as a “deploy key” on the server. More details later on this step.
upgrade.yaml – Upgrade all apt packages on remote server.

Push your Django project in a publicly accessible GIT repository

Probably you already have your project sources hosted on a public GIT repository like Github or Bitbucket. If you don’t, this is a perfect time to do it! I personally use Bitbucket, because it offers an unlimited number of private repositories.

In the following steps I assume that you have your Django project hosted on Bitbucket, under this path:

ssh://git@bitbucket.org/youruser/yourproject.git

Let the Ansible automation begins!

Inside the ansible directory, run the following command:

./ansible.sh system.yaml

You should see an output similar to this:

ok: [yourserver] => {
    "changed": false,
    "msg": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1kvkW9... ansible-generated on yourserver.example.com\n"
}

What you find in the msg variable is the public SSH key generated for the ubuntu user on remote server. You should copy the public key and add it as a “deploy key” in the settings of your GIT repository.

A deploy key is a read-only SSH key that will be used to clone your repository from the remote server. You can find more details in the Bitbucket and Github documentation.

Now you are ready to complete the deploy of your Django project! Run the following commands inside the ansible directory:

./ansible.sh packages.yaml
./ansible.sh postgresql.yaml
./ansible.sh config_files.yaml
./ansible.sh deploy.yaml

If all goes well you should be able to reach your Django project on your remote server public address, on port 80.

3. Update your Django project

Keeping your Django project updated on the remote server is very easy in this setup. You only need to push your changes to your GIT repository (on the master branch) and then you can run the following command inside the ansible directory:

./ansible.sh deploy.yaml

This playbook will perform the following tasks for you:

pull the updated master branch from your GIT repository;
eventually install new Python production requirements, and update the existing ones;
migrate the database to the latest version;
update your static files with collectstatic;
restart uwsgi.

4. Continuous delivery using Bitbucket Pipelines

In the follow up article Bitbucket Pipelines and Ansible: Continuous delivery for your Django project I explained how you can use Ansible together with Bitbucket Pipelines to implement a simple yet powerful continuous delivery setup for your Django project.

5. Conclusions

In this article I explained how to deploy a Django project in 15 minutes with Ansible. The key here is in following some Django best practices and leveraging the power on Ansible to quickly deploy a Django project on a remote server. I think that the time spent to learn and master those technologies is worth the effort for the time you’ll save in the future and all the human errors you’ll avoid by automating and standardizing your processes.

Of course this is only the tip of the iceberg of what you could do and automate with Ansible, but I found this simple approach to be already a big time saver in my daily work.

Have fun with Ansible!