ehmatthes.com

code, politics, and life

Simplifying Django deployments on Heroku

Posted at — Oct 19, 2020

What’s wrong with Heroku’s process?

I support a lot of beginners in deploying their first Django project to Heroku. It’s much simpler than deploying to a VPS such as Digital Ocean or Linode, but there are still a number of configuration steps that have to be done right, which seem like they could be automated.

Let’s say you have a relatively simple Django project which runs locally on the development server, using SQLite. Let’s also assume you’re using Git for version control, and that you already have the Heroku CLI package installed. What are the steps for an initial deployment? The Heroku docs tell you to create a Procfile, and install a number of packages that help make initial configurations for the database connection and static settings. Installing these packages can be confusing because the main recommended package, django-heroku, has been archived for a long time now. It has a dependency, psycopg2, which now requires that you have Postgres installed locally. So now we’re telling new Django developers who are using SQLite locally that they’ll have to install Postgres in order to push to Heroku, even if they won’t be using Postgres locally? There has to be a simpler way - one that’s friendlier to beginners, and simpler for experienced developers who want the initial deployment process to be as straightforward as possible.

A simpler process

This summer I made a modified version of the Heroku Python buildpack that automates as much of the deployment process as possible. The buildpack seems like the best place to make these configurations, because this is the code that’s examing the project as it’s being deployed. My goal in modifying the buildpack was to automate the initial configuration steps, but in a way that developers can then customize without having to undo any of the auto-configuration steps.

The modified buildpack is here. You can deploy a simple Django project using the following steps, from a terminal in the root directory of your project:

$ heroku login
$ heroku create
$ heroku buildpacks:set https://github.com/ehmatthes/heroku-buildpack-python.git
$ heroku config:set AUTOCONFIGURE_ALL=1
$ git push heroku main
$ heroku run python manage.py migrate
$ heroku open

Notice there are no modifications to the original Django project; everything that’s necessary for deployment is done on the server. Once you’ve run these commands, you should have a working project deployed to Heroku.

Note: Don’t use this for a production project. This is an experimental, demonstration project.

Technical details

If you’re familiar with the Heroku Python buildpack, you might be curious to know how this works. All of the changes happen in one script called autoconfigure. You can see this script here. The script is called from bin/compile.

The autoconfigure script does the following:

These are the same steps Heroku recommends people do on their own. It’s just done automatically from within the buildpack.

Why focus on the buildpack?

First of all everyone I’ve talked to, including Heroku staff, agrees that recommending an archived library is not a good approach. At this point ‘deprecated’ is probably a more accurate term than ‘archived’ for the django-heroku library, although ‘deprecated’ tends to imply that something newer and more up to date has taken the old library’s place.

The buildpack is where everything happens in a simple deployment. Consider static files, for example. The current recommended approach is to use Whitenoise. If that goes out of style and something better comes along, people who are configuring static files on their own would need to modify their project to use the new library. If this configuration is done inside the buildpack, the end user doesn’t need to do anything - the buildpack can just start using the newer library, and the end user just sees their project continue to be served correctly, static files included. This applies to all aspects of simple deployments.

Current limitations

Update 10/21/20: This buildpack should work for Pipenv-based projects as well as projects that use requirements.txt.

I believe this will only work for a project that uses a requirements.txt file. I don’t think it would take much work to make Pipenv-based projects work as well, but I haven’t used Pipenv much so I haven’t tried to do that work yet.

Try it out

If you have a small to medium size Django project that you can try deployment on, please give this process a try. I’d love to know what works and what doesn’t, so please report your results on the successes and failures issue.

If you don’t have a good project to try this with but you want to see how the process works, I’ve posted an instance of the Learning Log project from Python Crash Course as an example. There’s a Pipenv-based version here as well. Here are the steps to run the requirements.txt version of the project locally:

~$ git clone https://github.com/ehmatthes/learning_log_heroku_test.git
Cloning into 'learning_log_heroku_test'...
  ...
~$ cd learning_log_heroku_test/
~/learning_log_heroku_test$ python3 -m venv ll_env
~/learning_log_heroku_test$ source ll_env/bin/activate
(ll_env) ~/learning_log_heroku_test$ pip install --upgrade pip
  ...
Successfully installed pip-20.2.4
(ll_env) ~/learning_log_heroku_test$ pip install -r requirements.txt 
  ...
Installing collected packages: asgiref, soupsieve, beautifulsoup4, sqlparse, pytz, Django, django-bootstrap4
Successfully installed Django-3.0.6 asgiref-3.2.7 beautifulsoup4-4.9.1 django-bootstrap4-1.1.1 pytz-2020.1 soupsieve-2.0.1 sqlparse-0.3.1
(ll_env) ~/learning_log_heroku_test$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, learning_logs, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  ...
(ll_env) ~/learning_log_heroku_test$ python manage.py runserver
  ...
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

At this point you should have a running instance of the Learning Log project, accessible at localhost:8000. Register a user and make a topic and maybe an entry to satisfy yourself that the project is working fully.

Now you can deploy the project without making any local changes:

(ll_env) ~/learning_log_heroku_test$ heroku login
heroku: Press any key to open up the browser to login or q to exit: 
  ...
Logging in... done
(ll_env) ~/learning_log_heroku_test$ heroku create
Creating app... done, ⬢ still-crag-73341
https://still-crag-73341.herokuapp.com/ | https://git.heroku.com/still-crag-73341.git
(ll_env) ~/learning_log_heroku_test$ heroku buildpacks:set https://github.com/ehmatthes/heroku-buildpack-python.git
Buildpack set. Next release on still-crag-73341 will use https://github.com/ehmatthes/heroku-buildpack-python.git.
Run git push heroku main to create a new release using this buildpack.
(ll_env) ~/learning_log_heroku_test$ heroku config:set AUTOCONFIGURE_ALL=1
Setting AUTOCONFIGURE_ALL and restarting ⬢ still-crag-73341... done, v3
AUTOCONFIGURE_ALL: 1
(ll_env) ~/learning_log_heroku_test$ git push heroku main
Enumerating objects: 193, done.
  ...
remote: Building source:
remote: 
remote: -----> Python app detected
remote: *****> Auto-configuring this project.
remote:        Generated Procfile with following process:
remote:          web: gunicorn learning_log.wsgi --log-file -
remote:        Added gunicorn to requirements.txt.
remote:        Modifying settings.py:
remote:          Set ALLOWED_HOSTS = ['*'].
remote:        Configuring database
remote:          Adding psycopg2 to requirements.txt.
remote:          Adding dj-database-url to requirements.txt.
remote:          Modifying settings to use Heroku Postgres database.
remote:        Configuring static files.
remote:          Adding whitenoise to requirements.txt.
remote:          Configuring settings for static files.
remote:          Creating folder to store static files.
remote:        Finished auto-configure work.
remote: -----> Installing python-3.6.10
  ...
remote:        Successfully installed Django-3.0.6 asgiref-3.2.7 beautifulsoup4-4.9.1 dj-database-url-0.5.0 django-bootstrap4-1.1.1 gunicorn-20.0.4 psycopg2-2.8.6 pytz-2020.1 soupsieve-2.0.1 sqlparse-0.3.1 whitenoise-5.2.0
remote: -----> $ python manage.py collectstatic --noinput
remote:        130 static files copied to '/tmp/build_96d6343a/staticfiles'.
remote: 
remote: -----> Discovering process types
remote:        Procfile declares types -> web
remote: 
remote: -----> Compressing...
remote:        Done: 51.8M
remote: -----> Launching...
remote:        Released v6
remote:        https://still-crag-73341.herokuapp.com/ deployed to Heroku
remote: 
remote: Verifying deploy... done.
To https://git.heroku.com/still-crag-73341.git
 * [new branch]      main -> main
(ll_env) ~/learning_log_heroku_test$ heroku run python manage.py migrate
Running python manage.py migrate on ⬢ still-crag-73341... up, run.8242 (Free)
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, learning_logs, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  ...
  Applying learning_logs.0003_topic_owner... OK
  Applying sessions.0001_initial... OK
(ll_env) ~/learning_log_heroku_test$ heroku open

Now you should have a working version of the project running from a Heroku server. Register a user and add a topic to convince yourself that everything is working on the live project just as its working on the local projectt.

Again, notice that we haven’t done anything to change the local project; all we’ve done is issue some commands for Heroku to run on its own servers. You can see the auto-configuration work in the output starting at *****>, through the following ----->.

This means you can continue to develop as you have been locally, and still push your changes to Heroku without ever modifying your local project. When your program grows in complexity, you’ll need to learn more about specific aspects of the deployment process. But this approach should cover beginners deploying their first simple projects, and it will probably cover deployment of real-world projects that stay fairly simple over time as well.

Note: I destroyed this app after drafting this post, so you won’t find my instance of the Learning Log project running on Heroku.

Long term goals

I don’t want to maintain this project for long. It’s really Heroku’s job to simplify deployments as much as possible. I’m more than happy to show a proof-of-concept approach that I think Heroku should consider adopting, but I’m also quite wary of doing Heroku’s work for free on an ongoing basis.

If you like this approach please reach out to Heroku and ask them to consider adopting this approach, or a similar one that eliminates reliance on third-party projects to simplify the initial deployment process. My conversations with Heroku staff over the years have always been friendly, but I’ve also never gotten any sense of urgency on their part around this issue.

I may do a few things to provide a further proof of concept:

Feedback

I don’t have comments enabled at this time, but if you have any feedback feel free to open an issue, or reach out to me directly.