{"title":"Matt's Dev Blog - Django","link":[{"@attributes":{"href":"https:\/\/mattsegal.dev\/","rel":"alternate"}},{"@attributes":{"href":"https:\/\/mattsegal.dev\/feeds\/django.atom.xml","rel":"self"}}],"id":"https:\/\/mattsegal.dev\/","updated":"2022-01-13T12:00:00+11:00","entry":[{"title":"How to setup Django with Pytest on GitHub Actions","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/django-with-pytest-on-github-actions.html","rel":"alternate"}},"published":"2022-01-13T12:00:00+11:00","updated":"2022-01-13T12:00:00+11:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2022-01-13:\/django-with-pytest-on-github-actions.html","summary":"<p>Someone recently asked me<\/p>\n<blockquote>\n<p>When is a good time to get automated testing setup on a new Django project?<\/p>\n<\/blockquote>\n<p>The answer is \"now\". There are other good times, but now is best. In this post I'll briefly make my case for why, and show you an example of a minimal \u2026<\/p>","content":"<p>Someone recently asked me<\/p>\n<blockquote>\n<p>When is a good time to get automated testing setup on a new Django project?<\/p>\n<\/blockquote>\n<p>The answer is \"now\". There are other good times, but now is best. In this post I'll briefly make my case for why, and show you an example of a minimal setup of Django running tests with <a href=\"https:\/\/docs.pytest.org\/en\/6.2.x\/index.html\">pytest<\/a> with fully automated <a href=\"https:\/\/www.atlassian.com\/continuous-delivery\/continuous-integration\">continuous integration<\/a> (CI) using <a href=\"https:\/\/github.com\/features\/actions\">GitHub Actions<\/a>.<\/p>\n<p>As soon as you know a Django project is going to be \"serious\", then you should get it set up to run tests. So, potentially before you write any features. My approach is to get testing setup and to write a dummy test or two and then get it running in CI. This means that as soon as you start writing features then you will have everything you need to write a real test and have it run automatically on every commit.<\/p>\n<p>The alternate scenario is you start adding features and get swept up in that process. At some point you'll think \"hmm maybe I should write a test for this...\", but if you don't have tests and CI set up already then you're more likely to say \"nah, fuck it I'll do it later\" and not write the test. Getting pytest to work with Django on GitHub actions is pretty easy these days. Bite the bullet, it tastes better than you may expect.<\/p>\n<p>Or you could just not write any tests. This is fine for small personal projecs. Tests are a lot of things but they're not fun. For more serious endeavours though, not having tests will lead to riskier deployments, longer feedback loops on errors and less confidence in making big changes. Have you ever done a huge, wild refactor of a chunk of code, followed by a set of passing tests? It feels great man, that's when you're really living.<\/p>\n<p>The other question is: when should I run my tests? Sometimes you forget or you can't be bothered. This is where GitHub Actions (or any other CI) is very useful. You can set this service up to automatically run your tests <em>every time<\/em> you push a commit up to GitHub.<\/p>\n<p>Let's go then: how do you set up Django + pytest + GitHub Actions? All the code discussed here can be found in this <a href=\"https:\/\/github.com\/MattSegal\/django-pytest-github-actions\">example GitHub repository<\/a>.<\/p>\n<h2>Installation<\/h2>\n<p>Alongside Django you will need to install <a href=\"https:\/\/docs.pytest.org\/en\/6.2.x\/\"><code>pytest<\/code><\/a> and <a href=\"https:\/\/pytest-django.readthedocs.io\/en\/latest\/\"><code>pytest-django<\/code><\/a>. These libraries are not required to run tests with Django: the <a href=\"https:\/\/docs.djangoproject.com\/en\/4.0\/topics\/testing\/overview\/\">official docs<\/a> show you how to use Python's unittest library instead. I like pytest better though, and I think you will too. My <a href=\"https:\/\/github.com\/MattSegal\/django-pytest-github-actions\/blob\/master\/requirements.txt\">requirements.txt<\/a> file looks like this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>django\npytest\npytest-django\n<\/code><\/pre><\/div>\n\n<p>I don't pin my dependencies because I'm lazy: what can I say? I recommend you setup a <a href=\"https:\/\/realpython.com\/python-virtual-environments-a-primer\/\">virtual environment<\/a> and then install as follows:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>pip install -r requirements.txt\n<\/code><\/pre><\/div>\n\n<h2>Configuraton<\/h2>\n<p>You can configure pytest with a standard <a href=\"https:\/\/snarky.ca\/what-the-heck-is-pyproject-toml\/\">pyproject.toml<\/a> file. <a href=\"https:\/\/github.com\/MattSegal\/django-pytest-github-actions\/blob\/master\/app\/pyproject.toml\">Here's mine<\/a>. The most important thing is to set <a href=\"https:\/\/docs.djangoproject.com\/en\/4.0\/topics\/settings\/#envvar-DJANGO_SETTINGS_MODULE\"><code>DJANGO_SETTINGS_MODULE<\/code><\/a> so pytest knows which settings to use. It's good to have a separate set of test settings for your project so that you can avoid, for example, accidently changing your production environment with credentials stored in settings when you run a test.<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"k\">[tool.pytest.ini_options]<\/span><span class=\"w\"><\/span>\n<span class=\"na\">DJANGO_SETTINGS_MODULE<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">&quot;demo.settings&quot;<\/span><span class=\"w\"><\/span>\n<span class=\"na\">filterwarnings<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">[<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"na\">&quot;ignore::UserWarning&quot;,<\/span><span class=\"w\"><\/span>\n<span class=\"na\">]<\/span><span class=\"w\"><\/span>\n<\/code><\/pre><\/div>\n\n<p>This file should live in whichever folder you will be running <code>pytest<\/code> from. For the reference project, that means in the <code>.\/app<\/code> folder alongside <code>manage.py<\/code>.<\/p>\n<h2>Adding a dummy test<\/h2>\n<p>That's a good start. Now we can test the setup so far with a dummy test. This test does nothing: it always passes, but it verifies that all the plumbing is working. In pytest, tests are just functions that use assert statements to check things:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"k\">def<\/span> <span class=\"nf\">test_nothing<\/span><span class=\"p\">():<\/span>\n    <span class=\"sd\">&quot;&quot;&quot;A dummy test&quot;&quot;&quot;<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"kc\">True<\/span>\n<\/code><\/pre><\/div>\n\n<p>Pytest looks for a <code>tests<\/code> folder in your Django apps. For example, here is the <a href=\"https:\/\/github.com\/MattSegal\/django-pytest-github-actions\/tree\/master\/app\/web\/tests\">tests folder<\/a> in the reference project. So this dummy test function could live in a file named <code>app\/web\/tests\/test_dummy.py<\/code>. You can add as many tests to a file as you like, or have as many test files as you like. Avoid duplicate names though!<\/p>\n<h2>Running the tests locally<\/h2>\n<p>At this stage it's good to check that the dummy test works by running pytest from the command line:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>pytest -vv\n<\/code><\/pre><\/div>\n\n<p>Read <code>-vv<\/code> as \"very verbose\". Here are <a href=\"https:\/\/github.com\/MattSegal\/django-pytest-github-actions#running-tests\">specific instructions<\/a> for anyone trying out the reference project. Hopefully that worked. You may see a folder called <code>.pytest_cache<\/code> appear in your project. I recommend you <a href=\"https:\/\/www.atlassian.com\/git\/tutorials\/saving-changes\/gitignore\">gitignore<\/a> this.<\/p>\n<p>Now let's add some more meaningful example tests before we move on to setting up GitHub Actions.<\/p>\n<h2>Adding a basic view test<\/h2>\n<p>My reference project has a very basic view named \"goodbye\" which just returns the text \"Goodbye world\". Here it is:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"k\">def<\/span> <span class=\"nf\">goodbye_view<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">HttpResponse<\/span><span class=\"p\">(<\/span><span class=\"sa\">f<\/span><span class=\"s2\">&quot;Goodbye world&quot;<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n<p>You can test that this view returns the expected response using the <a href=\"https:\/\/docs.djangoproject.com\/en\/4.0\/topics\/testing\/tools\/#the-test-client\">Django test client<\/a>. Pytest has a handy feature called <a href=\"https:\/\/docs.pytest.org\/en\/6.2.x\/fixture.html\">fixtures<\/a>, which is a little piece of magic where you ask for an speficic object via the test function arguments and pytest automagically provides it. In this case we add \"client\" to the function arguments to get a test client. It's a little out of scope for this post, but you can write your own fixtures too!<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"k\">def<\/span> <span class=\"nf\">test_goodbye_view<\/span><span class=\"p\">(<\/span><span class=\"n\">client<\/span><span class=\"p\">):<\/span>\n    <span class=\"sd\">&quot;&quot;&quot;Test that goodbye view works&quot;&quot;&quot;<\/span>\n    <span class=\"c1\"># Build the URL from the url&#39;s name<\/span>\n    <span class=\"n\">url<\/span> <span class=\"o\">=<\/span> <span class=\"n\">reverse<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;goodbye&quot;<\/span><span class=\"p\">)<\/span>\n    <span class=\"c1\"># Make a GET request to the view using the test client<\/span>\n    <span class=\"n\">response<\/span> <span class=\"o\">=<\/span> <span class=\"n\">client<\/span><span class=\"o\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">(<\/span><span class=\"n\">url<\/span><span class=\"p\">)<\/span>\n    <span class=\"c1\"># Verify that the response is correct<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"n\">response<\/span><span class=\"o\">.<\/span><span class=\"n\">status_code<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">200<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"n\">response<\/span><span class=\"o\">.<\/span><span class=\"n\">content<\/span> <span class=\"o\">==<\/span> <span class=\"sa\">b<\/span><span class=\"s2\">&quot;Goodbye world&quot;<\/span>\n<\/code><\/pre><\/div>\n\n<p>Very nice, but you will find that you need to do a little more work to test views that include database queries.<\/p>\n<h2>Adding a view test with database interaction<\/h2>\n<p>With pytest-django you need to <em>explicitly<\/em> request access to the database using the <a href=\"https:\/\/pytest-django.readthedocs.io\/en\/latest\/helpers.html#pytest-mark-django-db-request-database-access\">pytest.mark.django_db<\/a> decorator. Below is an example of a test that hits the database. In this example there is a page view counter that increments +1 every time someone views the page:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"k\">def<\/span> <span class=\"nf\">hello_view<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">counter<\/span><span class=\"p\">,<\/span> <span class=\"n\">_<\/span> <span class=\"o\">=<\/span> <span class=\"n\">PageViewCount<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span><span class=\"o\">.<\/span><span class=\"n\">get_or_create<\/span><span class=\"p\">(<\/span><span class=\"n\">title<\/span><span class=\"o\">=<\/span><span class=\"s2\">&quot;hello&quot;<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">counter<\/span><span class=\"o\">.<\/span><span class=\"n\">count<\/span> <span class=\"o\">+=<\/span> <span class=\"mi\">1<\/span>\n    <span class=\"n\">counter<\/span><span class=\"o\">.<\/span><span class=\"n\">save<\/span><span class=\"p\">()<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">HttpResponse<\/span><span class=\"p\">(<\/span><span class=\"sa\">f<\/span><span class=\"s2\">&quot;Hello world. The counter is: <\/span><span class=\"si\">{<\/span><span class=\"n\">counter<\/span><span class=\"o\">.<\/span><span class=\"n\">count<\/span><span class=\"si\">}<\/span><span class=\"s2\">&quot;<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n<p>So if you load the page over and over again it should say:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>Hello world. The counter is: 1\nHello world. The counter is: 2\nHello world. The counter is: 3\nHello world. The counter is: 4\n... etc\n<\/code><\/pre><\/div>\n\n<p>Here is a test for this view:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"kn\">import<\/span> <span class=\"nn\">pytest<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">django.urls<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">reverse<\/span>\n\n<span class=\"kn\">from<\/span> <span class=\"nn\">web.models<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">PageViewCount<\/span>\n\n\n<span class=\"nd\">@pytest<\/span><span class=\"o\">.<\/span><span class=\"n\">mark<\/span><span class=\"o\">.<\/span><span class=\"n\">django_db<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">test_hello_view<\/span><span class=\"p\">(<\/span><span class=\"n\">client<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">url<\/span> <span class=\"o\">=<\/span> <span class=\"n\">reverse<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;hello&quot;<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"n\">PageViewCount<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span><span class=\"o\">.<\/span><span class=\"n\">count<\/span><span class=\"p\">()<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">0<\/span>\n\n    <span class=\"n\">response<\/span> <span class=\"o\">=<\/span> <span class=\"n\">client<\/span><span class=\"o\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">(<\/span><span class=\"n\">url<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"n\">response<\/span><span class=\"o\">.<\/span><span class=\"n\">status_code<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">200<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"n\">PageViewCount<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span><span class=\"o\">.<\/span><span class=\"n\">count<\/span><span class=\"p\">()<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">1<\/span>\n    <span class=\"n\">counter<\/span> <span class=\"o\">=<\/span> <span class=\"n\">PageViewCount<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span><span class=\"o\">.<\/span><span class=\"n\">last<\/span><span class=\"p\">()<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"n\">counter<\/span><span class=\"o\">.<\/span><span class=\"n\">count<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">1<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"sa\">b<\/span><span class=\"s2\">&quot;Hello world&quot;<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">response<\/span><span class=\"o\">.<\/span><span class=\"n\">content<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"sa\">b<\/span><span class=\"s2\">&quot;The counter is: 1&quot;<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">response<\/span><span class=\"o\">.<\/span><span class=\"n\">content<\/span>\n\n    <span class=\"n\">response<\/span> <span class=\"o\">=<\/span> <span class=\"n\">client<\/span><span class=\"o\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">(<\/span><span class=\"n\">url<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"n\">response<\/span><span class=\"o\">.<\/span><span class=\"n\">status_code<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">200<\/span>\n    <span class=\"n\">counter<\/span><span class=\"o\">.<\/span><span class=\"n\">refresh_from_db<\/span><span class=\"p\">()<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"n\">counter<\/span><span class=\"o\">.<\/span><span class=\"n\">count<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">2<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"sa\">b<\/span><span class=\"s2\">&quot;The counter is: 2&quot;<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">response<\/span><span class=\"o\">.<\/span><span class=\"n\">content<\/span>\n<\/code><\/pre><\/div>\n\n<h2>Setting up GitHub Actions<\/h2>\n<p>Ok so all our tests are running locally, how do we get them to run automatically in GitHub Actions? You can configure an action by adding a config file to your GitHub project at the location <code>.github\/workflows\/whatever.yml<\/code>. I named mine <a href=\"https:\/\/github.com\/MattSegal\/django-pytest-github-actions\/blob\/master\/.github\/workflows\/tests.yml\">tests.yml<\/a>.<\/p>\n<p>Let's walk through the contents of this file (docs <a href=\"https:\/\/docs.github.com\/en\/actions\">here<\/a>):<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># The name of the action<\/span><span class=\"w\"><\/span>\n<span class=\"nt\">name<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"l l-Scalar l-Scalar-Plain\">Django Tests<\/span><span class=\"w\"><\/span>\n<span class=\"c1\"># When the action is triggered<\/span><span class=\"w\"><\/span>\n<span class=\"nt\">on<\/span><span class=\"p\">:<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"nt\">push<\/span><span class=\"p\">:<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"nt\">branches<\/span><span class=\"p\">:<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"p p-Indicator\">-<\/span><span class=\"w\"> <\/span><span class=\"l l-Scalar l-Scalar-Plain\">master<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"nt\">pull_request<\/span><span class=\"p\">:<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"nt\">branches<\/span><span class=\"p\">:<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"p p-Indicator\">-<\/span><span class=\"w\"> <\/span><span class=\"l l-Scalar l-Scalar-Plain\">master<\/span><span class=\"w\"><\/span>\n\n<span class=\"c1\"># What to do when the action is triggered<\/span><span class=\"w\"><\/span>\n<span class=\"nt\">jobs<\/span><span class=\"p\">:<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"c1\"># A job called &#39;build&#39; - arbitrary<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"nt\">build<\/span><span class=\"p\">:<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"c1\"># Run on a Ubuntu VM<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"nt\">runs-on<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"l l-Scalar l-Scalar-Plain\">ubuntu-latest<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"nt\">steps<\/span><span class=\"p\">:<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"c1\"># Checkout the GitHub repo<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"p p-Indicator\">-<\/span><span class=\"w\"> <\/span><span class=\"nt\">uses<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"l l-Scalar l-Scalar-Plain\">actions\/checkout@v2<\/span><span class=\"w\"><\/span>\n\n<span class=\"w\">      <\/span><span class=\"c1\"># Install Python 3.8<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"p p-Indicator\">-<\/span><span class=\"w\"> <\/span><span class=\"nt\">name<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"l l-Scalar l-Scalar-Plain\">Set up Python 3.8<\/span><span class=\"w\"><\/span>\n<span class=\"w\">        <\/span><span class=\"nt\">uses<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"l l-Scalar l-Scalar-Plain\">actions\/setup-python@v2<\/span><span class=\"w\"><\/span>\n<span class=\"w\">        <\/span><span class=\"nt\">with<\/span><span class=\"p\">:<\/span><span class=\"w\"><\/span>\n<span class=\"w\">          <\/span><span class=\"nt\">python-version<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s\">&quot;3.8&quot;<\/span><span class=\"w\"><\/span>\n\n<span class=\"w\">      <\/span><span class=\"c1\"># Pip install project dependencies<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"p p-Indicator\">-<\/span><span class=\"w\"> <\/span><span class=\"nt\">name<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"l l-Scalar l-Scalar-Plain\">Install dependencies<\/span><span class=\"w\"><\/span>\n<span class=\"w\">        <\/span><span class=\"nt\">run<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p p-Indicator\">|<\/span><span class=\"w\"><\/span>\n<span class=\"w\">          <\/span><span class=\"no\">python -m pip install --upgrade pip<\/span><span class=\"w\"><\/span>\n<span class=\"w\">          <\/span><span class=\"no\">pip install -r requirements.txt<\/span><span class=\"w\"><\/span>\n\n<span class=\"w\">      <\/span><span class=\"c1\"># Move into the Django project folder (.\/app) and run pytest<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"p p-Indicator\">-<\/span><span class=\"w\"> <\/span><span class=\"nt\">name<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"l l-Scalar l-Scalar-Plain\">Test with pytest<\/span><span class=\"w\"><\/span>\n<span class=\"w\">        <\/span><span class=\"nt\">working-directory<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"l l-Scalar l-Scalar-Plain\">.\/app<\/span><span class=\"w\"><\/span>\n<span class=\"w\">        <\/span><span class=\"nt\">run<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"l l-Scalar l-Scalar-Plain\">pytest -vv<\/span><span class=\"w\"><\/span>\n<\/code><\/pre><\/div>\n\n<p>That's it, now pytest will run on every commit to master, and every pull request to master. You can see the actions for the reference project <a href=\"https:\/\/github.com\/MattSegal\/django-pytest-github-actions\/actions\">here<\/a>. Every test run will put a little tick or cross in your GitHub commit history.<\/p>\n<p><img alt=\"test ticks\" src=\"https:\/\/mattsegal.dev\/django-test-tick.png\"><\/p>\n<p>You can also embed a nice little badge in your README:<\/p>\n<p><a href=\"https:\/\/github.com\/MattSegal\/django-pytest-github-actions\/actions\/workflows\/tests.yml\"><img alt=\"Django Tests\" src=\"https:\/\/github.com\/MattSegal\/django-pytest-github-actions\/actions\/workflows\/tests.yml\/badge.svg\"><\/a><\/p>\n<h2>Conclusion<\/h2>\n<p>I hope this post helps you get started with writing and running automated tests for your Django project. They're a real lifesaver. If you liked this post about testing, you might also like this post about different testing styles (<a href=\"https:\/\/mattsegal.dev\/alternate-test-styles.html\">There's no one right way to test your code<\/a>) and this post about setting up pytest on GitHub actions, without Django (<a href=\"https:\/\/mattsegal.dev\/pytest-on-github-actions.html\">Run your Python unit tests via GitHub actions<\/a>).<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"My (free) Django monitoring stack for 2022","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/django-monitoring-stack.html","rel":"alternate"}},"published":"2022-01-01T12:00:00+11:00","updated":"2022-01-01T12:00:00+11:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2022-01-01:\/django-monitoring-stack.html","summary":"<p>You've built and deployed a website using Django. Congrats!\nAfter that initial high of successfully launching your site comes the grubby work of fixing bugs. There are so many things that <s>can<\/s> will go wrong.\nPages may crash with 500 errors in prod, but not locally. Some offline tasks never \u2026<\/p>","content":"<p>You've built and deployed a website using Django. Congrats!\nAfter that initial high of successfully launching your site comes the grubby work of fixing bugs. There are so many things that <s>can<\/s> will go wrong.\nPages may crash with 500 errors in prod, but not locally. Some offline tasks never finish. The site becomes <a href=\"https:\/\/twitter.com\/mattdsegal\/status\/1473462877772136448\">mysteriously unresponsive<\/a>. This one pain-in-the-ass user keeps complaining that file uploads \"don't work\"\nbut refuses to elaborate further: \"they just don't work okay!?!\".<\/p>\n<p>If enough issues crop up and you aren't able to solve them quickly and decisively, then you will lose the precious trust of your coworkers or clients. Often reputational damage isn't caused by the bug itself, but by the perception that you have no idea what's going on.<\/p>\n<p>Imagine that you are able to find out about bugs or outages <em>as they happen<\/em>. You proactively warn your users that the site is down, not the other way around. You can quickly reproduce problems locally and push a fix to prod in a matter of hours. Sounds good right? You're going to need a good \"monitoring stack\" to achieve this dream state of omniscient hyper-competence.<\/p>\n<p>You'll need a few different (free) tools to get a holistic picture of what your Django app is doing:<\/p>\n<ul>\n<li><strong>Uptime monitoring<\/strong>: tells you when the site is down (<a href=\"https:\/\/www.statuscake.com\/\">StatusCake<\/a>)<\/li>\n<li><strong>Error reporting<\/strong>: tells you when an application error occurs, collects details (<a href=\"https:\/\/sentry.io\/welcome\/\">Sentry<\/a>)<\/li>\n<li><strong>Log aggregation<\/strong>: allows you to read about what happened on your servers (<a href=\"https:\/\/www.sumologic.com\/\">Sumologic<\/a>)<\/li>\n<li><strong>Performance<\/strong>: tells you how long requests took, what's fast, what's slow (<a href=\"tps:\/\/sentry.io\/welcome\/\">Sentry<\/a>, <a href=\"https:\/\/newrelic.com\/products\/application-monitoring\">New Relic<\/a>)<\/li>\n<\/ul>\n<p>In the rest of this post I'll talk about these SaaS tools in more detail and why I like to use the ones linked above.<\/p>\n<h2>Uptime monitoring<\/h2>\n<p>It's quite embarrasing when your site goes down, but what's more embarrasing is when you learn about it from <em>someone else<\/em>. An uptime monitoring service can help: it sends a request to your site every few minutes and pings you (Slack, email) when it's unresponsive. This allows you to quickly get your site back online, hopefully before anyone notices. If you want to get fancy you can build a health check route (eg. <code>\/health-check\/<\/code>) into your Django app which, for example, checks that the database, or cache, or whatever are still online as well.<\/p>\n<p>Another benefit of uptime monitoring is that you'll get a clear picture of when the outage started. For example, in the picture below you can see that a website of mine stopped responding to requests between ~21:00 and ~23:30 UTC. You can use this knowledge of exactly <em>when<\/em> the site become unresponsive to check other sources of information, such as server logs or error reports for clues.<\/p>\n<p><img alt=\"downtime\" src=\"https:\/\/mattsegal.dev\/img\/downtime.png\"><\/p>\n<p>I like to use <a href=\"https:\/\/www.statuscake.com\/\">StatusCake<\/a> for this function because it's free, simple and easy to set up.<\/p>\n<h2>Error reporting<\/h2>\n<p>There are lots of ways for your site to break that don't render it completely unresponsive. A user might click a button to submit a form and receive a 500 error page because you made some trivial coding mistake that wasn't caught by your <a href=\"https:\/\/mattsegal.dev\/pytest-on-github-actions.html\">automated testing pipeline<\/a>. This user comes to you and complains that \"the site is broken\". Sometimes they will provide you with a very detailed explanation of what they did to produce the error, which you can use to replicate the issue, but as often as not they may, infuriated by your shitty website and seemingly antagonistic line of questioning, follow up with \"iTs JuST brOken OKAY!?\". Wouldn't it be nice to get the detailed information that you need to fix the bug without having to talk to a human?<\/p>\n<p>This is where error reporting comes in. When your Django web app catches some kind of exception, then an error reporting library can inspect the error and send the details to a SaaS service which records it for you. These error reporting tools capture heaps of useful information, such as:<\/p>\n<ul>\n<li>When the error happened first and most recently<\/li>\n<li>The exception type and message<\/li>\n<li>Which line of code triggered the error<\/li>\n<li>The stack trace of the error<\/li>\n<li>The value of local variables in each frame of the stack trace<\/li>\n<li>The Python version, package versions, user browser, IP, etc etc etc.<\/li>\n<\/ul>\n<p>This rich source of information makes error reporting a vital tool. It really shines when you encounter errors that <em>only<\/em> happen in production, where you have no idea how to replicate them locally. <a href=\"https:\/\/sentry.io\/welcome\/\">Sentry<\/a> is great for this task because it's free, easy to set up and has a great web UI. You can set up Sentry to send you error alerts via Slack and\/or email.<\/p>\n<h2>Log aggregation<\/h2>\n<p>Production errors can be more complicated than a simple Python exception crashing a page. Sometimes, much more complicated. If you want to get a feel for the twisted shit computers will get up to then give <a href=\"https:\/\/rachelbythebay.com\/w\/\">Rachel by the Bay<\/a> a read. To solve the trickier issues in production you're going to need to reconstruct what actually happened at the time of the error. You'll need to draw upon multiple sources of information, such as:<\/p>\n<ul>\n<li>application logs (eg. <a href=\"https:\/\/mattsegal.dev\/file-logging-django.html\">Django logs<\/a>)<\/li>\n<li>webserver logs (eg. <a href=\"https:\/\/mattsegal.dev\/django-gunicorn-nginx-logging.html\">NGINX, Gunicorn logs<\/a>)<\/li>\n<li>logs from other services (eg. Postgres, syslog, etc)<\/li>\n<\/ul>\n<p>You can <code>ssh<\/code> into your server and read these logs from the command line using <code>less<\/code> or <code>grep<\/code> or <code>awk<\/code> or something. Even so, it's much more convenient to access these logs via a log aggregation service's web UI, where you can run search queries to quickly find the log lines of interest. These tools work by running a \"logging agent\" on your server, which watches files of interest and sends them to a centralised server.<\/p>\n<p><img alt=\"logging\" src=\"https:\/\/mattsegal.dev\/img\/logging.png\"><\/p>\n<p>This model is paritcularly valuable if you have transient infrastructure (servers that don't last forever) or if you have many different servers, or if you want to limit <code>ssh<\/code> access for security reasons.<\/p>\n<p><a href=\"https:\/\/www.sumologic.com\/\">Sumologic<\/a> if my favourite free SaaS for this task because it's easy to install the logging agent and add new files to be watched. The search is pretty good as well. The main downside is that web UI can be a little complicated and overwhelming at times. The search DSL is very powerful but I always need to look up the syntax. Log retention times seem reasonable, 30 days by default. The Sumologic agent seems to consume several hundred MB of RAM (~300MB?).<\/p>\n<p><a href=\"https:\/\/www.papertrail.com\/\">Papertrail<\/a> is, in my opinion, worse than Sumologic in every way I can think of. However, it is also free and presents a simple web UI for viewing and searching your logs. If you're interested I wrote about setting up Papertrail <a href=\"https:\/\/mattsegal.dev\/django-logging-papertrail.html\">here<\/a>. <a href=\"https:\/\/docs.newrelic.com\/docs\/logs\/get-started\/get-started-log-management\/\">New Relic<\/a> offer a logging service as well - never tried it though. There are open source logging solutions like <a href=\"https:\/\/www.elastic.co\/\">Elasticsearch<\/a> + Kibana and other alternatives, but they come with the downside of having to run them yourself: \"now you have two problems\".<\/p>\n<h2>Performance montioring<\/h2>\n<p>Sometimes your website isn't broken per-se, but it's too slow. People hate slow websites. You can often diagnose and fix these issues locally using tools like <a href=\"https:\/\/django-debug-toolbar.readthedocs.io\/en\/latest\/\">Django Debug Toolbar<\/a> (I made a video on how to do this <a href=\"https:\/\/mattsegal.dev\/django-debug-toolbar-performance.html\">here<\/a>), but sometimes the slowness only happens in production. Furthermore, riffing on the general theme of this article, you want to know about (and fix) slow pages before your boss walks over to your desk and complains about it.<\/p>\n<p>Performance monitoring tools instrument your Django web app and record information about how long various requests take. What's fast? What's slow? Which pages have problems? I recommend that you start out by using <a href=\"https:\/\/sentry.io\/welcome\/\">Sentry<\/a> for this task because their performance monitoring service comes bundled with their error reporting by default. It's kind of basic, but maybe that's all you need.<\/p>\n<p>The best appilcation performance monitoring for Django that I know of is <a href=\"https:\/\/newrelic.com\/products\/application-monitoring\">New Relic's offering<\/a>, which seems to have a free tier. The request traces that they track include a very detailed breakdown of <em>where<\/em> the time was spent in serving a request. For example, it will tell you how much time was spent querying the database, or a cache, or building HTML templates. Sometimes you need that level of detail to solve tricky performance issues. The downside of using New Relic is that you have to reconfigure your app server to boot using their <a href=\"https:\/\/docs.newrelic.com\/docs\/apm\/agents\/python-agent\/\">agent<\/a> as a wrapper.<\/p>\n<p>Although it's not strictly on-topic, <a href=\"https:\/\/pagespeed.web.dev\/\">PageSpeed Insights<\/a> is pretty useful for checking page load performance from a front-end perspective. If you're interested in more on Django web app performance then you might like this post I wrote, where I ponder: <a href=\"https:\/\/mattsegal.dev\/is-django-too-slow.html\">is Django too slow?<\/a><\/p>\n<h2>Conclusion<\/h2>\n<p>This list is not exhaustive or definitive, it's just the free-tier tools that I like to use for my freelance and personal projects.\nNevertheless I hope you find them useful.\nIt can be a pain to integrate them all into your app, but over the long run they'll save you a lot of time and energy.<\/p>\n<p>Be prepared!<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"How to setup Django with React","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/django-react.html","rel":"alternate"}},"published":"2020-10-24T12:00:00+11:00","updated":"2020-10-24T12:00:00+11:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-10-24:\/django-react.html","summary":"<p>It's not too hard to get started with either Django or React. Both have great documentation and there are lots of tutorials online. \nThe tricky part is getting them to work together. Many people start with a Django project and then decide that they want to \"add React\" to it \u2026<\/p>","content":"<p>It's not too hard to get started with either Django or React. Both have great documentation and there are lots of tutorials online. \nThe tricky part is getting them to work together. Many people start with a Django project and then decide that they want to \"add React\" to it.\nHow do you do that though? Popular React scaffolding tools like <a href=\"https:\/\/github.com\/facebook\/create-react-app\">Create React App<\/a> don't offer you a clear way to integrate with Django, leaving you to figure it out yourself. Even worse, there isn't just one way to set up a Django\/React project. There are dozens of <a href=\"https:\/\/mattsegal.dev\/django-spa-infrastructure.html\">possible methods<\/a>, each with different pros and cons. Every time I create a new project using these tools I find the options overwhelming.<\/p>\n<p>I think that most people should start with a setup that is as close to vanilla Django as possible: you take your existing Django app and sprinkle a little React on it to make the frontend more dynamic and interactive. For most cases, creating a completely seperate \"single page app\" frontend creates a lot of complexity and challenges without providing very much extra value for you or your users.<\/p>\n<p>In this series of posts I will present an opinionated guide on how to setup and deploy a Django\/React webapp. The focus will be on keeping things simple, incremental and understanding each step. I want you to be in a position to debug any problems yourself. At the end of each post, you should have a working project that you can use.<\/p>\n<p>I'm going to assume that you know:<\/p>\n<ul>\n<li>the <a href=\"https:\/\/developer.mozilla.org\/en-US\/docs\/Learn\/Getting_started_with_the_web\">basics of web development<\/a> (HTML, CSS, JavaScript)<\/li>\n<li>the <a href=\"https:\/\/docs.djangoproject.com\/en\/3.1\/intro\/tutorial01\/\">basics of Django<\/a> (views, templates, static files)<\/li>\n<li>the <a href=\"https:\/\/reactjs.org\/tutorial\/tutorial.html\">basics of React<\/a> (components, props, rendering)<\/li>\n<\/ul>\n<p>I'm <strong>not<\/strong> going to assume that you know anything about Webpack, Babel, or any other JavaScript toolchain insanity.<\/p>\n<h2>Example project<\/h2>\n<p>The example code for this guide is hosted on <a href=\"https:\/\/github.com\/MattSegal\/django-react-guide\">this GitHub repo<\/a>. The code for each section is available as a Git branch:<\/p>\n<ul>\n<li><a href=\"https:\/\/github.com\/MattSegal\/django-react-guide\/tree\/part-1-initial-django\">Starting point<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/MattSegal\/django-react-guide\/tree\/part-2-add-webpack\">Adding Webpack<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/MattSegal\/django-react-guide\/tree\/part-3-add-babel-and-react\">Adding Babel and React<\/a><\/li>\n<\/ul>\n<p>Before you start the rest of the guide, I recommend setting up the example project by cloning the repo and following the instructions in the <a href=\"https:\/\/github.com\/MattSegal\/django-react-guide\/blob\/part-1-initial-django\/README.md\">README<\/a>:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>git clone https:\/\/github.com\/MattSegal\/django-react-guide.git\n<\/code><\/pre><\/div>\n\n<div class=\"loom-embed\"><iframe src=\"https:\/\/www.loom.com\/embed\/d238b8eb58dd44c89af7a4e3dd0c42a1\" frameborder=\"0\" webkitallowfullscreen mozallowfullscreen allowfullscreen style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%;\"><\/iframe><\/div>\n\n<h2>Django and static files<\/h2>\n<p>Before we dig into React, Babel and Webpack, I want to make sure that we have a common understanding around how static files work in Django:<\/p>\n<p><img alt=\"views and static files\" src=\"https:\/\/mattsegal.dev\/views-static.png\"><\/p>\n<p>The approach of this guide will be to re-use a lot of this existing setup. We will create an additional that system inserts our React app's JavaScript into a Django static files folder.<\/p>\n<p><img alt=\"views and static files plus mystery system\" src=\"https:\/\/mattsegal.dev\/views-static-mystery.png\"><\/p>\n<h2>Why can't we just write React in a single static file?<\/h2>\n<p>Why do we need to add a new system? Django is pretty complicated already. Can't we just write our React app in a single JavaScript file like you usually do when writing JavaScript for webpages? The answer is yes, you totally can! You can write a complete React app in a single HTML file:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"p\">&lt;<\/span><span class=\"nt\">html<\/span><span class=\"p\">&gt;<\/span>\n<span class=\"p\">&lt;<\/span><span class=\"nt\">body<\/span><span class=\"p\">&gt;<\/span>\n  <span class=\"cm\">&lt;!-- React mount point --&gt;<\/span>\n  <span class=\"p\">&lt;<\/span><span class=\"nt\">div<\/span> <span class=\"na\">id<\/span><span class=\"o\">=<\/span><span class=\"s\">&quot;app&quot;<\/span><span class=\"p\">&gt;&lt;\/<\/span><span class=\"nt\">div<\/span><span class=\"p\">&gt;<\/span>\n  <span class=\"cm\">&lt;!-- Download React library scripts --&gt;<\/span>\n  <span class=\"p\">&lt;<\/span><span class=\"nt\">script<\/span> <span class=\"na\">crossorigin<\/span> <span class=\"na\">src<\/span><span class=\"o\">=<\/span><span class=\"s\">&quot;https:\/\/unpkg.com\/react@16\/umd\/react.development.js&quot;<\/span><span class=\"p\">&gt;&lt;\/<\/span><span class=\"nt\">script<\/span><span class=\"p\">&gt;<\/span>\n  <span class=\"p\">&lt;<\/span><span class=\"nt\">script<\/span> <span class=\"na\">crossorigin<\/span> <span class=\"na\">src<\/span><span class=\"o\">=<\/span><span class=\"s\">&quot;https:\/\/unpkg.com\/react-dom@16\/umd\/react-dom.development.js&quot;<\/span><span class=\"p\">&gt;&lt;\/<\/span><span class=\"nt\">script<\/span><span class=\"p\">&gt;<\/span>\n  <span class=\"p\">&lt;<\/span><span class=\"nt\">script<\/span><span class=\"p\">&gt;<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"c1\">\/\/ Define the React app<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">App<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"p\">()<\/span><span class=\"w\"> <\/span><span class=\"p\">=&gt;<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"nx\">count<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"nx\">setCount<\/span><span class=\"p\">]<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"nx\">React<\/span><span class=\"p\">.<\/span><span class=\"nx\">useState<\/span><span class=\"p\">(<\/span><span class=\"mf\">0<\/span><span class=\"p\">)<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">onClick<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"p\">()<\/span><span class=\"w\"> <\/span><span class=\"p\">=&gt;<\/span><span class=\"w\"> <\/span><span class=\"nx\">setCount<\/span><span class=\"p\">(<\/span><span class=\"nx\">c<\/span><span class=\"w\"> <\/span><span class=\"p\">=&gt;<\/span><span class=\"w\"> <\/span><span class=\"nx\">c<\/span><span class=\"w\"> <\/span><span class=\"o\">+<\/span><span class=\"w\"> <\/span><span class=\"mf\">1<\/span><span class=\"p\">)<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"k\">return<\/span><span class=\"w\"> <\/span><span class=\"nx\">React<\/span><span class=\"p\">.<\/span><span class=\"nx\">createElement<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;div&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"kc\">null<\/span><span class=\"p\">,<\/span><span class=\"w\"><\/span>\n<span class=\"w\">        <\/span><span class=\"nx\">React<\/span><span class=\"p\">.<\/span><span class=\"nx\">createElement<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;h1&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"kc\">null<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;The count is &#39;<\/span><span class=\"w\"> <\/span><span class=\"o\">+<\/span><span class=\"w\"> <\/span><span class=\"nx\">count<\/span><span class=\"p\">),<\/span><span class=\"w\"><\/span>\n<span class=\"w\">        <\/span><span class=\"nx\">React<\/span><span class=\"p\">.<\/span><span class=\"nx\">createElement<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;button&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\"> <\/span><span class=\"nx\">onClick<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"nx\">onClick<\/span><span class=\"w\"> <\/span><span class=\"p\">},<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;Count&#39;<\/span><span class=\"p\">),<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"p\">)<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"p\">}<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"c1\">\/\/ Mount the app to the mount point.<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">root<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"nb\">document<\/span><span class=\"p\">.<\/span><span class=\"nx\">getElementById<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;app&#39;<\/span><span class=\"p\">)<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"nx\">ReactDOM<\/span><span class=\"p\">.<\/span><span class=\"nx\">render<\/span><span class=\"p\">(<\/span><span class=\"nx\">React<\/span><span class=\"p\">.<\/span><span class=\"nx\">createElement<\/span><span class=\"p\">(<\/span><span class=\"nx\">App<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"kc\">null<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"kc\">null<\/span><span class=\"p\">),<\/span><span class=\"w\"> <\/span><span class=\"nx\">root<\/span><span class=\"p\">)<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"p\">&lt;\/<\/span><span class=\"nt\">script<\/span><span class=\"p\">&gt;<\/span>\n<span class=\"p\">&lt;\/<\/span><span class=\"nt\">body<\/span><span class=\"p\">&gt;<\/span>\n<span class=\"p\">&lt;\/<\/span><span class=\"nt\">html<\/span><span class=\"p\">&gt;<\/span>\n<\/code><\/pre><\/div>\n\n<p>Why don't we just do this? There are a few issues with this approach of writing React apps:<\/p>\n<ul>\n<li>We can't use <a href=\"https:\/\/reactjs.org\/docs\/introducing-jsx.html\">JSX<\/a> syntax in our JavaScript<\/li>\n<li>It's harder to break our JavaScript code up into modules<\/li>\n<li>It's harder to install\/use external libraries<\/li>\n<\/ul>\n<div class=\"loom-embed\"><iframe src=\"https:\/\/www.loom.com\/embed\/8f2c4c6448144246b25beed21a7b4712\" frameborder=\"0\" webkitallowfullscreen mozallowfullscreen allowfullscreen style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%;\"><\/iframe><\/div>\n\n<h2>Webpack<\/h2>\n<p>The example code for this section <a href=\"https:\/\/github.com\/MattSegal\/django-react-guide\/tree\/part-1-initial-django\">starts here<\/a> and <a href=\"https:\/\/github.com\/MattSegal\/django-react-guide\/tree\/part-2-add-webpack\">ends here<\/a>.<\/p>\n<p>We need a tool that helps us use JSX, and it would be nice to also have a \"module bundling system\" which lets us install 3rd party libraries and split our JavaScript code up into lots of little files. For this purpose, we're going to use <a href=\"https:\/\/webpack.js.org\/\">Webpack<\/a>. Webpack is going to take our code, plus any 3rd party libraries that we want to install and combine them into a single JS file.<\/p>\n<p><img alt=\"webpack\" src=\"https:\/\/mattsegal.dev\/webpack.png\"><\/p>\n<p>In this step we will just to create a minimal working Webpack setup. We're not goint try to use React yet. By the end of this section, we won't have added any new JavaScript features, but Webpack will be working.<\/p>\n<p>To use Webpack you need to first install <a href=\"https:\/\/nodejs.org\/en\/\">NodeJS<\/a> so that you can run JavaScript outside of your web browser. You need to be able to run <code>node<\/code> and <code>npm<\/code> (the Node Package Manager) before you can continue.<\/p>\n<p>First, go into the example project and create a new folder called <code>frontend<\/code>.\nWe'll start by just copying over the existing JavaScript that is used by the Django app in <a href=\"https:\/\/github.com\/MattSegal\/django-react-guide\/blob\/part-1-initial-django\/backend\/todos\/static\/todos\/main.js\">main.js<\/a>. We're going to copy this into a \"source code\" folder at <code>frontend\/src\/index.js<\/code>.<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\">\/\/ frontend\/src\/index.js<\/span><span class=\"w\"><\/span>\n<span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">btn<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"nb\">document<\/span><span class=\"p\">.<\/span><span class=\"nx\">getElementById<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;click&#39;<\/span><span class=\"p\">)<\/span><span class=\"w\"><\/span>\n<span class=\"nx\">btn<\/span><span class=\"p\">.<\/span><span class=\"nx\">addEventListener<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;click&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"p\">()<\/span><span class=\"w\"> <\/span><span class=\"p\">=&gt;<\/span><span class=\"w\"> <\/span><span class=\"nx\">alert<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;You clicked the button!&#39;<\/span><span class=\"p\">))<\/span><span class=\"w\"><\/span>\n<\/code><\/pre><\/div>\n\n<p>Inside of the <code>frontend<\/code> folder, install Webpack using <code>npm<\/code> as follows:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>npm init --yes\nnpm install webpack webpack-cli\n<\/code><\/pre><\/div>\n\n<p>Now is a good time to update your <code>.gitignore<\/code> file to exclude <code>node_modules<\/code>. Next, we need to add a file that tells Webpack what to do, which is called <code>webpack.config.js<\/code><\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\">\/\/ frontend\/webpack.config.js<\/span><span class=\"w\"><\/span>\n<span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">path<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"nx\">require<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;path&#39;<\/span><span class=\"p\">)<\/span><span class=\"w\"><\/span>\n<span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">webpack<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"nx\">require<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;webpack&#39;<\/span><span class=\"p\">)<\/span><span class=\"w\"><\/span>\n<span class=\"nx\">module<\/span><span class=\"p\">.<\/span><span class=\"nx\">exports<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"c1\">\/\/ Where Webpack looks to load your JavaScript<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"nx\">entry<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"nx\">main<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"nx\">path<\/span><span class=\"p\">.<\/span><span class=\"nx\">resolve<\/span><span class=\"p\">(<\/span><span class=\"nx\">__dirname<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;src\/index.js&#39;<\/span><span class=\"p\">),<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"p\">},<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"nx\">mode<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;development&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"c1\">\/\/ Where Webpack spits out the results (the myapp static folder)<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"nx\">output<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"nx\">path<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"nx\">path<\/span><span class=\"p\">.<\/span><span class=\"nx\">resolve<\/span><span class=\"p\">(<\/span><span class=\"nx\">__dirname<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;..\/backend\/myapp\/static\/myapp\/&#39;<\/span><span class=\"p\">),<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"nx\">filename<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;[name].js&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"p\">},<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"nx\">plugins<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"c1\">\/\/ Don&#39;t output new files if there is an error<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"ow\">new<\/span><span class=\"w\"> <\/span><span class=\"nx\">webpack<\/span><span class=\"p\">.<\/span><span class=\"nx\">NoEmitOnErrorsPlugin<\/span><span class=\"p\">(),<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"p\">],<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"c1\">\/\/ Where find modules that can be imported (eg. React) <\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"nx\">resolve<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"nx\">extensions<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s1\">&#39;*&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;.js&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;.jsx&#39;<\/span><span class=\"p\">],<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"nx\">modules<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"w\"><\/span>\n<span class=\"w\">        <\/span><span class=\"nx\">path<\/span><span class=\"p\">.<\/span><span class=\"nx\">resolve<\/span><span class=\"p\">(<\/span><span class=\"nx\">__dirname<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;src&#39;<\/span><span class=\"p\">),<\/span><span class=\"w\"><\/span>\n<span class=\"w\">        <\/span><span class=\"nx\">path<\/span><span class=\"p\">.<\/span><span class=\"nx\">resolve<\/span><span class=\"p\">(<\/span><span class=\"nx\">__dirname<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;node_modules&#39;<\/span><span class=\"p\">),<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"p\">],<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"p\">},<\/span><span class=\"w\"><\/span>\n<span class=\"p\">}<\/span><span class=\"w\"><\/span>\n<\/code><\/pre><\/div>\n\n<p>Finally let's make it easy to run Webpack by including an entry in the \"scripts\" section of our <code>package.json<\/code> file:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\">\/\/ frontend\/package.json<\/span><span class=\"w\"><\/span>\n<span class=\"p\">{<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"c1\">\/\/ ...<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"s2\">&quot;scripts&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"s2\">&quot;dev&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;webpack --watch --config webpack.config.js&quot;<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"p\">},<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"c1\">\/\/ ...<\/span><span class=\"w\"><\/span>\n<span class=\"p\">}<\/span><span class=\"w\"><\/span>\n<\/code><\/pre><\/div>\n\n<p>The <code>--watch<\/code> flag is particularly useful: it makes Webpack re-run automatically on file change. Now we can run Webpack using <code>npm<\/code>:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>npm run dev\n<\/code><\/pre><\/div>\n\n<p>You will now see that the contents of your <code>main.js<\/code> file has been replaced with a crazy looking <code>eval<\/code> statement. If you check your Django app at <code>http:\/\/localhost:8000<\/code> you'll see that the JavaScript on the page still works, but it's now using the Webpack build output at <code>http:\/\/localhost:8000\/static\/myapp\/main.js<\/code>  <\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\">\/\/ backend\/myapp\/static\/myapp\/main.js<\/span><span class=\"w\"><\/span>\n<span class=\"nb\">eval<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;const btn = document.getElementById(&#39;click&#39;)\\nbtn.addEventListener(&#39;click&#39;, () =&gt; alert(&#39;You clicked the button!&#39;))\\n\\n\\n\/\/# sourceURL=webpack:\/\/frontend\/.\/src\/index.js?&quot;<\/span><span class=\"p\">);<\/span><span class=\"w\"><\/span>\n<\/code><\/pre><\/div>\n\n<p>This file is the Webpack build output. Webpack has taken our source file (<code>index.js<\/code>) and transformed it into an output file (<code>main.js<\/code>): <\/p>\n<p><img alt=\"webpack minimal\" src=\"https:\/\/mattsegal.dev\/webpack-minimal.png\"><\/p>\n<p>So now we have Webpack working. It's not doing anything particularly useful or interesting yet, but all the plumbing has been set up.<\/p>\n<div class=\"loom-embed\"><iframe src=\"https:\/\/www.loom.com\/embed\/b3dd1325841646a491728c1478a173d3\" frameborder=\"0\" webkitallowfullscreen mozallowfullscreen allowfullscreen style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%;\"><\/iframe><\/div>\n\n<h2>Source code vs. build outputs<\/h2>\n<p>It's a common newbie mistake to add Webpack build outputs like <code>main.js<\/code> to source control. It's a mistake because source control is for \"source code\", not \"build artifacts\". A build artifact is a file created by a build or compliation process. The reason you don't add build artifacts is because they're redundant: they are fully defined by the source code, so adding them just bloats the repo without adding any extra information. Even worse, having a mismatch between source code and build artifacts can create nasty errors that are hard to find. Some examples of build artifacts:<\/p>\n<ul>\n<li>Python bytecode (.pyc) file,s which are built from .py files by the Python interpeter<\/li>\n<li>.NET bytecode (.dll) files, built from compiling C# code<\/li>\n<li>Executable (.exe) files, build from compiling C code<\/li>\n<\/ul>\n<p>None of these things should go in source control unless there's a special reason to keep them. In general they should be kept out of Git using the <code>.gitignore<\/code> file.<\/p>\n<p>My approach for this project is to create a special Webpack-only folder in Django's static file called \"build\", which is ignored by Git.\nTo achieve this, you need to update your <code>webpack.config.js<\/code> file:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\">\/\/ frontend\/webpack.config.js<\/span><span class=\"w\"><\/span>\n<span class=\"c1\">\/\/ ...<\/span><span class=\"w\"><\/span>\n<span class=\"nx\">module<\/span><span class=\"p\">.<\/span><span class=\"nx\">exports<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"c1\">\/\/ ...<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"nx\">output<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"nx\">path<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"nx\">path<\/span><span class=\"p\">.<\/span><span class=\"nx\">resolve<\/span><span class=\"p\">(<\/span><span class=\"nx\">__dirname<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;..\/backend\/myapp\/static\/myapp\/build\/&#39;<\/span><span class=\"p\">),<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"nx\">filename<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;[name].js&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"p\">},<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"c1\">\/\/ ...<\/span><span class=\"w\"><\/span>\n<span class=\"p\">}<\/span><span class=\"w\"><\/span>\n<\/code><\/pre><\/div>\n\n<p>You will need to restart Webpack for these changes to take effect. Then you can add <code>build\/<\/code> to your <code>.gitignore<\/code> file.\nFinally, you will need to update the static file link in your Django template:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"cm\">&lt;!-- backend\/myapp\/templates\/myapp\/index.html --&gt;<\/span>\n<span class=\"p\">&lt;<\/span><span class=\"nt\">script<\/span> <span class=\"na\">src<\/span><span class=\"o\">=<\/span><span class=\"s\">&quot;{% static &#39;myapp\/build\/main.js&#39; %}&quot;<\/span><span class=\"p\">&gt;&lt;\/<\/span><span class=\"nt\">script<\/span><span class=\"p\">&gt;<\/span>\n<\/code><\/pre><\/div>\n\n<div class=\"loom-embed\"><iframe src=\"https:\/\/www.loom.com\/embed\/86893cc2f3c14a41ab347bc912678ec9\" frameborder=\"0\" webkitallowfullscreen mozallowfullscreen allowfullscreen style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%;\"><\/iframe><\/div>\n\n<h2>Adding React<\/h2>\n<p>The example code for this section <a href=\"https:\/\/github.com\/MattSegal\/django-react-guide\/tree\/part-2-add-webpack\">starts here<\/a> and <a href=\"https:\/\/github.com\/MattSegal\/django-react-guide\/tree\/part-3-add-babel-and-react\">ends here<\/a>.<\/p>\n<p>Now that Webpack is working, we can add React. Let's start by installing React in our <code>frontend<\/code> folder:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>npm install react react-dom\n<\/code><\/pre><\/div>\n\n<p>Now we can use React in our JavaScript source code. Let's re-use the small counter app I created earlier:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\">\/\/ frontend\/src\/index.js<\/span>\n<span class=\"k\">import<\/span> <span class=\"nx\">React<\/span> <span class=\"nx\">from<\/span> <span class=\"s\">&#39;react&#39;<\/span>\n<span class=\"k\">import<\/span> <span class=\"nx\">ReactDOM<\/span> <span class=\"nx\">from<\/span> <span class=\"s\">&#39;react-dom&#39;<\/span>\n\n<span class=\"c1\">\/\/ Define the React app<\/span>\n<span class=\"kd\">const<\/span> <span class=\"nx\">App<\/span> <span class=\"o\">=<\/span> <span class=\"p\">()<\/span> <span class=\"o\">=&gt;<\/span> <span class=\"p\">{<\/span>\n  <span class=\"kd\">const<\/span> <span class=\"p\">[<\/span><span class=\"nx\">count<\/span><span class=\"p\">,<\/span> <span class=\"nx\">setCount<\/span><span class=\"p\">]<\/span> <span class=\"o\">=<\/span> <span class=\"nx\">React<\/span><span class=\"p\">.<\/span><span class=\"nx\">useState<\/span><span class=\"p\">(<\/span><span class=\"m\">0<\/span><span class=\"p\">)<\/span>\n  <span class=\"kd\">const<\/span> <span class=\"nx\">onClick<\/span> <span class=\"o\">=<\/span> <span class=\"p\">()<\/span> <span class=\"o\">=&gt;<\/span> <span class=\"nx\">setCount<\/span><span class=\"p\">(<\/span><span class=\"nx\">c<\/span> <span class=\"o\">=&gt;<\/span> <span class=\"nx\">c<\/span> <span class=\"o\">+<\/span> <span class=\"m\">1<\/span><span class=\"p\">)<\/span>\n  <span class=\"k\">return<\/span> <span class=\"nx\">React<\/span><span class=\"p\">.<\/span><span class=\"nx\">createElement<\/span><span class=\"p\">(<\/span><span class=\"s\">&#39;div&#39;<\/span><span class=\"p\">,<\/span> <span class=\"k\">null<\/span><span class=\"p\">,<\/span>\n    <span class=\"nx\">React<\/span><span class=\"p\">.<\/span><span class=\"nx\">createElement<\/span><span class=\"p\">(<\/span><span class=\"s\">&#39;h1&#39;<\/span><span class=\"p\">,<\/span> <span class=\"k\">null<\/span><span class=\"p\">,<\/span> <span class=\"s\">&#39;The count is &#39;<\/span> <span class=\"o\">+<\/span> <span class=\"nx\">count<\/span><span class=\"p\">),<\/span>\n    <span class=\"nx\">React<\/span><span class=\"p\">.<\/span><span class=\"nx\">createElement<\/span><span class=\"p\">(<\/span><span class=\"s\">&#39;button&#39;<\/span><span class=\"p\">,<\/span> <span class=\"p\">{<\/span> <span class=\"nx\">onClick<\/span><span class=\"p\">:<\/span> <span class=\"nx\">onClick<\/span> <span class=\"p\">},<\/span> <span class=\"s\">&#39;Count&#39;<\/span><span class=\"p\">),<\/span>\n  <span class=\"p\">)<\/span>\n<span class=\"p\">}<\/span>\n<span class=\"c1\">\/\/ Mount the app to the mount point.<\/span>\n<span class=\"kd\">const<\/span> <span class=\"nx\">root<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">document<\/span><span class=\"p\">.<\/span><span class=\"nx\">getElementById<\/span><span class=\"p\">(<\/span><span class=\"s\">&#39;app&#39;<\/span><span class=\"p\">)<\/span>\n<span class=\"nx\">ReactDOM<\/span><span class=\"p\">.<\/span><span class=\"nx\">render<\/span><span class=\"p\">(<\/span><span class=\"nx\">React<\/span><span class=\"p\">.<\/span><span class=\"nx\">createElement<\/span><span class=\"p\">(<\/span><span class=\"nx\">App<\/span><span class=\"p\">,<\/span> <span class=\"k\">null<\/span><span class=\"p\">,<\/span> <span class=\"k\">null<\/span><span class=\"p\">),<\/span> <span class=\"nx\">root<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n<p>Now if you go to <code>http:\/\/localhost:8000\/<\/code> you should see a simple counter. If you inspect the contents of <code>main.js<\/code> at <code>http:\/\/localhost:8000\/static\/myapp\/build\/main.js<\/code>, you'll see that there is a <em>lot<\/em> more stuff included in the file. This is because Webpack has bundled up our code plus the development versions of React and ReactDOM into a single file:<\/p>\n<p><img alt=\"webpack\" src=\"https:\/\/mattsegal.dev\/webpack.png\"><\/p>\n<div class=\"loom-embed\"><iframe src=\"https:\/\/www.loom.com\/embed\/76bf5c576ff148aea4e0d332507ec381\" frameborder=\"0\" webkitallowfullscreen mozallowfullscreen allowfullscreen style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%;\"><\/iframe><\/div>\n\n<div class=\"ui divider\" style=\"margin: 1.5em 0;\"><\/div>\n<form action=\"https:\/\/dev.us19.list-manage.com\/subscribe\/post?u=e7a1ec466f7bb1732dbd23fc7&amp;id=ec345473bd\" method=\"post\" name=\"mc-embedded-subscribe-form\" target=\"_blank\" style=\"text-align: center; padding-bottom: 1em;\" novalidate>\n  <h3 class=\"subscribe-cta\">Get alerted when I publish new blog posts<\/h3>\n  <div class=\"ui fluid action input subscribe\">\n    <input\n      type=\"email\"\n      value=\"\"\n      name=\"EMAIL\"\n      placeholder=\"Enter your email address\"\n    \/>\n    <button class=\"ui primary button\" type=\"submit\" name=\"subscribe\">\n      Subscribe\n    <\/button>\n  <\/div>\n  <div style=\"position: absolute; left: -5000px;\" aria-hidden=\"true\">\n    <input\n      type=\"text\"\n      name=\"b_e7a1ec466f7bb1732dbd23fc7_ec345473bd\"\n      tabindex=\"-1\"\n      value=\"\"\n    \/>\n  <\/div>\n<\/form>\n<div class=\"ui divider\" style=\"margin: 1.5em 0;\"><\/div>\n\n<h2>Adding Babel<\/h2>\n<p>Next we need at tool that lets us write JSX. We want to be able to write our React components like this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"kd\">const<\/span> <span class=\"nx\">App<\/span> <span class=\"o\">=<\/span> <span class=\"p\">()<\/span> <span class=\"o\">=&gt;<\/span> <span class=\"p\">{<\/span>\n  <span class=\"kd\">const<\/span> <span class=\"p\">[<\/span><span class=\"nx\">count<\/span><span class=\"p\">,<\/span> <span class=\"nx\">setCount<\/span><span class=\"p\">]<\/span> <span class=\"o\">=<\/span> <span class=\"nx\">React<\/span><span class=\"p\">.<\/span><span class=\"nx\">useState<\/span><span class=\"p\">(<\/span><span class=\"m\">0<\/span><span class=\"p\">)<\/span>\n  <span class=\"kd\">const<\/span> <span class=\"nx\">onClick<\/span> <span class=\"o\">=<\/span> <span class=\"p\">()<\/span> <span class=\"o\">=&gt;<\/span> <span class=\"nx\">setCount<\/span><span class=\"p\">(<\/span><span class=\"nx\">c<\/span> <span class=\"o\">=&gt;<\/span> <span class=\"nx\">c<\/span> <span class=\"o\">+<\/span> <span class=\"m\">1<\/span><span class=\"p\">)<\/span>\n  <span class=\"k\">return<\/span> <span class=\"p\">(<\/span>\n    <span class=\"p\">&lt;<\/span><span class=\"nt\">div<\/span><span class=\"p\">&gt;<\/span>\n      <span class=\"p\">&lt;<\/span><span class=\"nt\">h1<\/span><span class=\"p\">&gt;<\/span>The count is <span class=\"p\">{<\/span><span class=\"nx\">count<\/span><span class=\"p\">}&lt;\/<\/span><span class=\"nt\">h1<\/span><span class=\"p\">&gt;<\/span>\n      <span class=\"p\">&lt;<\/span><span class=\"nt\">button<\/span> <span class=\"na\">onClick<\/span><span class=\"o\">=<\/span><span class=\"p\">{<\/span><span class=\"nx\">onClick<\/span><span class=\"p\">}&gt;<\/span>Count<span class=\"p\">&lt;\/<\/span><span class=\"nt\">button<\/span><span class=\"p\">&gt;<\/span>\n    <span class=\"p\">&lt;\/<\/span><span class=\"nt\">div<\/span><span class=\"p\">&gt;<\/span>\n  <span class=\"p\">)<\/span>\n<span class=\"p\">}<\/span>\n<\/code><\/pre><\/div>\n\n<p>and then some magic tool transforms it into regular JavaScript, like this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"kd\">const<\/span> <span class=\"nx\">App<\/span> <span class=\"o\">=<\/span> <span class=\"p\">()<\/span> <span class=\"o\">=&gt;<\/span> <span class=\"p\">{<\/span>\n  <span class=\"kd\">const<\/span> <span class=\"p\">[<\/span><span class=\"nx\">count<\/span><span class=\"p\">,<\/span> <span class=\"nx\">setCount<\/span><span class=\"p\">]<\/span> <span class=\"o\">=<\/span> <span class=\"nx\">React<\/span><span class=\"p\">.<\/span><span class=\"nx\">useState<\/span><span class=\"p\">(<\/span><span class=\"m\">0<\/span><span class=\"p\">)<\/span>\n  <span class=\"kd\">const<\/span> <span class=\"nx\">onClick<\/span> <span class=\"o\">=<\/span> <span class=\"p\">()<\/span> <span class=\"o\">=&gt;<\/span> <span class=\"nx\">setCount<\/span><span class=\"p\">(<\/span><span class=\"nx\">c<\/span> <span class=\"o\">=&gt;<\/span> <span class=\"nx\">c<\/span> <span class=\"o\">+<\/span> <span class=\"m\">1<\/span><span class=\"p\">)<\/span>\n  <span class=\"k\">return<\/span> <span class=\"nx\">React<\/span><span class=\"p\">.<\/span><span class=\"nx\">createElement<\/span><span class=\"p\">(<\/span><span class=\"s\">&#39;div&#39;<\/span><span class=\"p\">,<\/span> <span class=\"k\">null<\/span><span class=\"p\">,<\/span>\n    <span class=\"nx\">React<\/span><span class=\"p\">.<\/span><span class=\"nx\">createElement<\/span><span class=\"p\">(<\/span><span class=\"s\">&#39;h1&#39;<\/span><span class=\"p\">,<\/span> <span class=\"k\">null<\/span><span class=\"p\">,<\/span> <span class=\"s\">&#39;The count is &#39;<\/span> <span class=\"o\">+<\/span> <span class=\"nx\">count<\/span><span class=\"p\">),<\/span>\n    <span class=\"nx\">React<\/span><span class=\"p\">.<\/span><span class=\"nx\">createElement<\/span><span class=\"p\">(<\/span><span class=\"s\">&#39;button&#39;<\/span><span class=\"p\">,<\/span> <span class=\"p\">{<\/span> <span class=\"nx\">onClick<\/span><span class=\"p\">:<\/span> <span class=\"nx\">onClick<\/span> <span class=\"p\">},<\/span> <span class=\"s\">&#39;Count&#39;<\/span><span class=\"p\">),<\/span>\n  <span class=\"p\">)<\/span>\n<span class=\"p\">}<\/span>\n<\/code><\/pre><\/div>\n\n<p>That magic tool is <a href=\"https:\/\/babeljs.io\/\">Babel<\/a>, a JavaScript compiler that can transform JSX into standard JavaScript.\nBabel can use <a href=\"https:\/\/babeljs.io\/docs\/en\/plugins\">plugins<\/a>, which apply custom transforms to your source code.\nIt also offers <a href=\"https:\/\/babeljs.io\/docs\/en\/presets\">presets<\/a>, which are groups of plugins that work well together to achieve a goal.<\/p>\n<p>Now we're going to install a whole bunch of Babel stuff with <code>npm<\/code>:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>npm install --save-dev babel-loader @babel\/core @babel\/preset-react\n<\/code><\/pre><\/div>\n\n<p>What the hell is all of this? Let me break it down for you:<\/p>\n<ul>\n<li><strong><a href=\"https:\/\/babeljs.io\/docs\/en\/babel-core\">@babel\/core<\/a><\/strong>: The main Babel compiler library<\/li>\n<li><strong><a href=\"https:\/\/babeljs.io\/docs\/en\/babel-preset-react\">@babel\/preset-react<\/a><\/strong>: A collection of React plugins: tranforms JSX to regular JavaScript<\/li>\n<li><strong><a href=\"https:\/\/github.com\/babel\/babel-loader\">babel-loader<\/a><\/strong>: Allows Webpack to use Babel<\/li>\n<\/ul>\n<p>These are not the only Babel plugins that I like to use, but I didn't want to add too many new things at once.\nIn addition to installing the plugins\/presets, we need to tell Babel to use them, which we do with a config file called <code>.babelrc<\/code>.<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\">\/\/ frontend\/.babelrc<\/span><span class=\"w\"><\/span>\n<span class=\"p\">{<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"s2\">&quot;presets&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s2\">&quot;@babel\/preset-react&quot;<\/span><span class=\"p\">]<\/span><span class=\"w\"><\/span>\n<span class=\"p\">}<\/span><span class=\"w\"><\/span>\n<\/code><\/pre><\/div>\n\n<p>Next, we need to tell Webpack to use our new Babel compiler for all our JavaScript files:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\">\/\/ frontend\/webpack.config.js<\/span><span class=\"w\"><\/span>\n<span class=\"c1\">\/\/ ...<\/span><span class=\"w\"><\/span>\n<span class=\"nx\">module<\/span><span class=\"p\">.<\/span><span class=\"nx\">exports<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"c1\">\/\/ ...<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"c1\">\/\/ Add a rule so Webpack reads JS with Babel<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"nx\">module<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\"> <\/span><span class=\"nx\">rules<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"p\">{<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"nx\">test<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"sr\">\/\\.js$\/<\/span><span class=\"p\">,<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"nx\">exclude<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"sr\">\/node_modules\/<\/span><span class=\"p\">,<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"nx\">use<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s1\">&#39;babel-loader&#39;<\/span><span class=\"p\">],<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"p\">},<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"p\">]},<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"c1\">\/\/ ...<\/span><span class=\"w\"><\/span>\n<\/code><\/pre><\/div>\n\n<p>Essentially, this config change tells Webpack: \"for any file ending with <code>.js<\/code>, use <code>babel-loader<\/code> on that file, expect for anything in <code>node_modules<\/code>\".\nFinally, we can now use JSX in our React app:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\">\/\/ frontend\/src\/index.js<\/span>\n<span class=\"k\">import<\/span> <span class=\"nx\">React<\/span> <span class=\"nx\">from<\/span> <span class=\"s\">&#39;react&#39;<\/span>\n<span class=\"k\">import<\/span> <span class=\"nx\">ReactDOM<\/span> <span class=\"nx\">from<\/span> <span class=\"s\">&#39;react-dom&#39;<\/span>\n\n<span class=\"c1\">\/\/ Define the React app<\/span>\n<span class=\"kd\">const<\/span> <span class=\"nx\">App<\/span> <span class=\"o\">=<\/span> <span class=\"p\">()<\/span> <span class=\"o\">=&gt;<\/span> <span class=\"p\">{<\/span>\n  <span class=\"kd\">const<\/span> <span class=\"p\">[<\/span><span class=\"nx\">count<\/span><span class=\"p\">,<\/span> <span class=\"nx\">setCount<\/span><span class=\"p\">]<\/span> <span class=\"o\">=<\/span> <span class=\"nx\">React<\/span><span class=\"p\">.<\/span><span class=\"nx\">useState<\/span><span class=\"p\">(<\/span><span class=\"m\">0<\/span><span class=\"p\">)<\/span>\n  <span class=\"kd\">const<\/span> <span class=\"nx\">onClick<\/span> <span class=\"o\">=<\/span> <span class=\"p\">()<\/span> <span class=\"o\">=&gt;<\/span> <span class=\"nx\">setCount<\/span><span class=\"p\">(<\/span><span class=\"nx\">c<\/span> <span class=\"o\">=&gt;<\/span> <span class=\"nx\">c<\/span> <span class=\"o\">+<\/span> <span class=\"m\">1<\/span><span class=\"p\">)<\/span>\n  <span class=\"k\">return<\/span> <span class=\"p\">(<\/span>\n    <span class=\"p\">&lt;<\/span><span class=\"nt\">div<\/span><span class=\"p\">&gt;<\/span>\n      <span class=\"p\">&lt;<\/span><span class=\"nt\">h1<\/span><span class=\"p\">&gt;<\/span>The count is <span class=\"p\">{<\/span><span class=\"nx\">count<\/span><span class=\"p\">}&lt;\/<\/span><span class=\"nt\">h1<\/span><span class=\"p\">&gt;<\/span>\n      <span class=\"p\">&lt;<\/span><span class=\"nt\">button<\/span> <span class=\"na\">onClick<\/span><span class=\"o\">=<\/span><span class=\"p\">{<\/span><span class=\"nx\">onClick<\/span><span class=\"p\">}&gt;<\/span>Count<span class=\"p\">&lt;\/<\/span><span class=\"nt\">button<\/span><span class=\"p\">&gt;<\/span>\n    <span class=\"p\">&lt;\/<\/span><span class=\"nt\">div<\/span><span class=\"p\">&gt;<\/span>\n  <span class=\"p\">)<\/span>\n<span class=\"p\">}<\/span>\n<span class=\"c1\">\/\/ Mount the app to the mount point.<\/span>\n<span class=\"kd\">const<\/span> <span class=\"nx\">root<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">document<\/span><span class=\"p\">.<\/span><span class=\"nx\">getElementById<\/span><span class=\"p\">(<\/span><span class=\"s\">&#39;app&#39;<\/span><span class=\"p\">)<\/span>\n<span class=\"nx\">ReactDOM<\/span><span class=\"p\">.<\/span><span class=\"nx\">render<\/span><span class=\"p\">(&lt;<\/span><span class=\"nt\">App<\/span> <span class=\"p\">\/&gt;,<\/span> <span class=\"nx\">root<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n<p>You will need to restart Webpack for the config changes to be loaded. After that, you should be able to visit <code>http:\/\/localhost:8000\/<\/code> and view your counter app, now working with JSX.<\/p>\n<div class=\"loom-embed\"><iframe src=\"https:\/\/www.loom.com\/embed\/18e5b20ee31344b588aa17dd902344ce\" frameborder=\"0\" webkitallowfullscreen mozallowfullscreen allowfullscreen style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%;\"><\/iframe><\/div>\n\n<h2>Deployment<\/h2>\n<p>I won't cover deployment in detail in this post, because it's long enough already, but in short, you can now deploy your Django\/React app as follows:<\/p>\n<ul>\n<li>Install JavaScript dependencies with <code>npm<\/code><\/li>\n<li>Run Webpack to create build artifacts in your Django static files<\/li>\n<li>Deploy Django how you normally would<\/li>\n<\/ul>\n<p>There a few things that it would be good to change before deploying, like not using \"development\" mode in Webpack, but this workflow should get you started for now.\nIf you have never deployed a Django app before, I've written an <a href=\"https:\/\/mattsegal.dev\/simple-django-deployment.html\">introductory guide<\/a> on that as well, which uses the same incremental, explanation-heavy style as this guide.<\/p>\n<h2>Next steps<\/h2>\n<p>There is a <strong>lot<\/strong> of stuff I didn't cover in this guide, which I'd like to write about in the future. Here are some things that I didn't cover, which are important or useful when building a React\/Django app:<\/p>\n<ul>\n<li>Hot reloading<\/li>\n<li>Deployment<\/li>\n<li>Passing requests\/data between Django and React<\/li>\n<li>Modular CSS \/ SCSS \/ styled components<\/li>\n<li>Routing and code-splitting<\/li>\n<li>Authentication<\/li>\n<\/ul>","category":{"@attributes":{"term":"Django"}}},{"title":"A Django project blueprint to help you learn by doing","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/django-survey-project.html","rel":"alternate"}},"published":"2020-10-03T12:00:00+10:00","updated":"2020-10-03T12:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-10-03:\/django-survey-project.html","summary":"<p>There's an awkward point when you're learning Django where you've done the <a href=\"https:\/\/docs.djangoproject.com\/en\/3.1\/intro\/install\/\">official tutorial<\/a> and maybe built a simple project, like a to-do list, and now you want to try something a little more advanced. People say that you should \"learn by building things\", which is good advice, but it \u2026<\/p>","content":"<p>There's an awkward point when you're learning Django where you've done the <a href=\"https:\/\/docs.djangoproject.com\/en\/3.1\/intro\/install\/\">official tutorial<\/a> and maybe built a simple project, like a to-do list, and now you want to try something a little more advanced. People say that you should \"learn by building things\", which is good advice, but it leaves you unsure about what to <em>actually build<\/em>. <\/p>\n<p>In this post I'll share two things:<\/p>\n<ul>\n<li>a description of a Django project for beginners, which you can build; and<\/li>\n<li>a short guide on how to design a new website from scratch<\/li>\n<\/ul>\n<p>I won't introduce many new tools or technical concepts beyond what is already in the Django tutorial. The project can be built using just the basic Django features. There is no need to use REST Framework, JavaScript, React, Webpack, Babel, JSON or AJAX to get this done. Only Django, HTML and CSS are required.<\/p>\n<p>Even though this project only uses simple tools, I think building it is worthwhile for a beginner, since it will introduce you to many of the common themes of backend web development.<\/p>\n<h1>Project overview<\/h1>\n<p>In this project, you will build a Django app that runs a survey website. On this site, users can create surveys and send them out to other people to get answers. A user can sign up, create a survey and add multi-choice questions to it. They can then send a survey link to other people, who will answer all the questions. The user who created the survey can see how many people answered, and what percentage of people chose each multi-choice option.<\/p>\n<p>That's the whole app.  I have created a <a href=\"https:\/\/github.com\/MattSegal\/django-survey\">reference implementation on my GitHub<\/a> which you can look at if you get stuck when building it yourself.<\/p>\n<p>The project description sounds simple, doesn't it? I thought this would take me 8 hours to design and build, but I spent <strong>20 hours<\/strong> at the keyboard to get it done. Software projects are hard to estimate before they are built, since they have a <a href=\"http:\/\/johnsalvatier.org\/blog\/2017\/reality-has-a-surprising-amount-of-detail\">surprising amount of detail<\/a> that you don't think about beforehand.<\/p>\n<h1>Designing the app<\/h1>\n<p>So now you know what you're building, but you're not ready to write any code yet. We need to create a design first. As the saying goes: <em>weeks of coding can save hours of planning<\/em>. <\/p>\n<p>This design will have three parts:<\/p>\n<ul>\n<li><strong>User journeys<\/strong>: where you decide who is using your app and how they will use it<\/li>\n<li><strong>Data models<\/strong>: where you decide how you will structure the database<\/li>\n<li><strong>Webpage wireframes<\/strong>: where you decide what your user interface (UI) will look like<\/li>\n<\/ul>\n<h1>User journey<\/h1>\n<p>The most important thing to do when building a website is to consider the users and their goals. In this case, I think there are two sets of users:<\/p>\n<ul>\n<li><strong>Survey takers<\/strong>: people who want to answer a survey's questions<\/li>\n<li><strong>Survey creators<\/strong>: people who want to create a survey, send it out and view the answers<\/li>\n<\/ul>\n<p>To better understand who your users are and what they want, you should construct a <a href=\"https:\/\/en.wikipedia.org\/wiki\/User_journey\">user journey<\/a> for each of them: a high-level description of the steps that they will need to take to get what they want. This is easily represented as a diagram, created with a free wireframing tool like <a href=\"https:\/\/excalidraw.com\/\">Exalidraw<\/a> or <a href=\"https:\/\/wireflow.co\/\">Wireflow<\/a>.<\/p>\n<p>Let's start with the person who is answering the survey, the \"survey taker\", who has a simple user journey:<\/p>\n<p><img alt=\"user journey for survey taker\" src=\"https:\/\/mattsegal.dev\/journey-taker.png\"><\/p>\n<p>Next, let's look at the person who created the survey, the \"survey creator\":<\/p>\n<p><img alt=\"user journey for survey creator\" src=\"https:\/\/mattsegal.dev\/journey-creator.png\"><\/p>\n<p>Creating these diagrams will force you to think about what you will need to build and why. For example, a survey creator will probably need a user account and the ability to \"log in\", since they will want private access to their surveys. Lots of thoughts about how to build your app will cross your mind when you are mapping these user journeys.<\/p>\n<h1>Data models<\/h1>\n<p>Once you know what your users want to do, you should focus on what data you will need to describe all of the things in your app. So far we have vague ideas of \"surveys\", \"questions\", \"answers\" and \"results\", but we need a more specific description of these things so that we can write our Model classes in Django.<\/p>\n<p>To better understand your data, I recommend that you create a simple diagram that displays your models and how they relate to each other. Each connection between a model is some kind of foreign key relation. Something like this:<\/p>\n<p><img alt=\"app data model\" src=\"https:\/\/mattsegal.dev\/data-model.png\"><\/p>\n<p>I explain how I came up with this particular data model in this <a href=\"https:\/\/mattsegal.dev\/django-survey-project-data-model.html\">appendix page<\/a>.<\/p>\n<p>You don't need to get too formal or technical with these diagrams. They're just a starting point, not a perfect, final description of how your app will work. Also, the data model which I made isn't the only possible one for this app. Feel free to make your own and do it differently.<\/p>\n<div class=\"ui divider\" style=\"margin: 1.5em 0;\"><\/div>\n<form action=\"https:\/\/dev.us19.list-manage.com\/subscribe\/post?u=e7a1ec466f7bb1732dbd23fc7&amp;id=ec345473bd\" method=\"post\" name=\"mc-embedded-subscribe-form\" target=\"_blank\" style=\"text-align: center; padding-bottom: 1em;\" novalidate>\n  <h3 class=\"subscribe-cta\">Get alerted when I publish new blog posts<\/h3>\n  <div class=\"ui fluid action input subscribe\">\n    <input\n      type=\"email\"\n      value=\"\"\n      name=\"EMAIL\"\n      placeholder=\"Enter your email address\"\n    \/>\n    <button class=\"ui primary button\" type=\"submit\" name=\"subscribe\">\n      Subscribe\n    <\/button>\n  <\/div>\n  <div style=\"position: absolute; left: -5000px;\" aria-hidden=\"true\">\n    <input\n      type=\"text\"\n      name=\"b_e7a1ec466f7bb1732dbd23fc7_ec345473bd\"\n      tabindex=\"-1\"\n      value=\"\"\n    \/>\n  <\/div>\n<\/form>\n<div class=\"ui divider\" style=\"margin: 1.5em 0;\"><\/div>\n\n<h1>Webpage wireframes<\/h1>\n<p>Now we have an idea of how our users will interact with the app and we know how we will structure our data. Next, we design our user interfaces. I suggest you create a rough <a href=\"https:\/\/www.usability.gov\/how-to-and-tools\/methods\/wireframing.html\">wireframe<\/a> that describes the user interface for each webpage. Creating wireframes for webpages is a good idea for two reasons:<\/p>\n<ul>\n<li>Wireframing allows you to <strong>quickly<\/strong> explore different page designs and it forces you to think about how your app needs to work<\/li>\n<li>It's <strong>much<\/strong> easier to write HTML and CSS for pages where you already have a simple design to work from<\/li>\n<\/ul>\n<p>You can use a free wireframing tool like <a href=\"https:\/\/excalidraw.com\/\">Exalidraw<\/a> or <a href=\"https:\/\/wireflow.co\/\">Wireflow<\/a> for these diagrams. Keep in mind that this project doesn't use JavaScript, so you can't get too fancy with custom interactions. You will need to use  <a href=\"https:\/\/developer.mozilla.org\/en-US\/docs\/Learn\/Forms\">HTML forms<\/a> to POST data to the backend.<\/p>\n<p>You can create your own wireframes or you can use the ones that I've already created, which are all listed in this <a href=\"https:\/\/mattsegal.dev\/django-survey-project-wireframes.html\">appendix page<\/a> with some additional notes for each page:<\/p>\n<ul>\n<li><a href=\"https:\/\/mattsegal.dev\/django-survey-project-wireframes.html#start\">Starting the survey<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/django-survey-project-wireframes.html#answer\">Answering the survey<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/django-survey-project-wireframes.html#submit\">Survey submitted<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/django-survey-project-wireframes.html#landing\">Landing page<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/django-survey-project-wireframes.html#signup\">Signing up<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/django-survey-project-wireframes.html#login\">Logging in<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/django-survey-project-wireframes.html#list\">Survey list<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/django-survey-project-wireframes.html#create\">Create a survey<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/django-survey-project-wireframes.html#edit\">Edit a survey<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/django-survey-project-wireframes.html#addquestion\">Add questions to a survey<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/django-survey-project-wireframes.html#addoption\">Add options to a survey question<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/django-survey-project-wireframes.html#details\">Survey details<\/a><\/li>\n<\/ul>\n<h1>General advice<\/h1>\n<p>Now with some user journeys, a data model and a set of wireframes, you should be ready to start building your Django app. This project blueprint will help you get started, but there is still a lot of work for you to do if you want to build this app. You still need to:<\/p>\n<ul>\n<li>decide on a URL schema<\/li>\n<li>create models to represent the data<\/li>\n<li>create forms to validate the user-submitted data<\/li>\n<li>write HTML templates to build each page<\/li>\n<li>add views to bind everything together<\/li>\n<\/ul>\n<p>There's about 12 views, 12 templates, 5 forms and 5 models to write. Given all this work, it's really important that you <strong>focus<\/strong> and keep the scope of this project narrow. Keep everything <strong>simple<\/strong>. Don't use any JavaScript and write as little CSS as possible. Use a CSS framework like <a href=\"https:\/\/getbootstrap.com\/docs\/4.0\/getting-started\/introduction\/\">Boostrap<\/a> or <a href=\"https:\/\/semantic-ui.com\/\">Semantic UI<\/a> if you want it to look nice.  Get something simple working <strong>first<\/strong>, and then you can make it fancy later. If you don't focus, you could spend weeks or months on this project before it's done.<\/p>\n<p>As a specific example, consider the user authentication feature. In this app, your users can log in or sign up. To really make the auth system \"complete\", you could also add a log out button, a password reset page, and an email validation feature. I think you should skip these features for now though, and get the core functionality working first.<\/p>\n<p>Software projects are never finished, and you can improve this app again and again even after you are \"done\". Don't try to make it perfect, just finish it.<\/p>\n<h1>Next steps<\/h1>\n<p>I hope you find this blueprint project and design guide helpful. If you actually end up building this, send me an email! I'd love to see it. If you like this post and you want to read some more stuff I've written about Django, check out:<\/p>\n<ul>\n<li><a href=\"https:\/\/mattsegal.dev\/simple-django-deployment.html\">A beginner's guide to Django deployment<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/how-to-read-django-docs.html\">How to read the Django documentation<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/django-portable-setup.html\">How to make your Django project easy to move and share <\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/github-resume-polish.html\">How to polish your GitHub projects when you're looking for a job<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/django-debug-tips.html\">Tips for debugging with Django<\/a><\/li>\n<\/ul>\n<p>You can also subscribe to my mailing list below for emails when I post new articles.<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"How to use both camelCase and snake_case in your frontend and backend","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/camel-and-snake-case.html","rel":"alternate"}},"published":"2020-09-24T12:00:00+10:00","updated":"2020-09-24T12:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-09-24:\/camel-and-snake-case.html","summary":"<p>Python uses <code>snake_case<\/code> variable naming while JavaScript favours <code>camelCase<\/code>. \nWhen you're buiding an web API with Django then you'll be using both langauges together. How do you keep your styles consistent? You <em>could<\/em> just use one style for both your frontend and backend, but it looks ugly. Perhaps this \u2026<\/p>","content":"<p>Python uses <code>snake_case<\/code> variable naming while JavaScript favours <code>camelCase<\/code>. \nWhen you're buiding an web API with Django then you'll be using both langauges together. How do you keep your styles consistent? You <em>could<\/em> just use one style for both your frontend and backend, but it looks ugly. Perhaps this is not the biggest problem in your life right now, but it's a nice one to solve and it's easy to fix.<\/p>\n<p>In this post I'll show you can use snake case on the backend and camel case on the frontend, with the help of the the <code>camelize<\/code> and <code>snakeize<\/code> JS libraries.<\/p>\n<h3>The problem: out of place naming styles<\/h3>\n<p>Let's say you've got some Django code that presents an API for a <code>Person<\/code> model:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># Inside your Django app.<\/span>\n<span class=\"c1\"># The data model<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">Person<\/span><span class=\"p\">(<\/span><span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">Model<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">full_name<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">CharField<\/span><span class=\"p\">(<\/span><span class=\"n\">max_length<\/span><span class=\"o\">=<\/span><span class=\"mi\">64<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">biggest_problem<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">CharField<\/span><span class=\"p\">(<\/span><span class=\"n\">max_length<\/span><span class=\"o\">=<\/span><span class=\"mi\">128<\/span><span class=\"p\">)<\/span>\n\n<span class=\"c1\"># The serializer<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">PersonSerializer<\/span><span class=\"p\">(<\/span><span class=\"n\">serializers<\/span><span class=\"o\">.<\/span><span class=\"n\">ModelSerializer<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">class<\/span> <span class=\"nc\">Meta<\/span><span class=\"p\">:<\/span>\n        <span class=\"n\">model<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Person<\/span>\n        <span class=\"n\">fields<\/span> <span class=\"o\">=<\/span> <span class=\"p\">[<\/span><span class=\"s2\">&quot;full_name&quot;<\/span><span class=\"p\">,<\/span> <span class=\"s2\">&quot;biggest_problem&quot;<\/span><span class=\"p\">]<\/span>\n\n<span class=\"c1\"># The API view<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">PersonViewSet<\/span><span class=\"p\">(<\/span><span class=\"n\">viewsets<\/span><span class=\"o\">.<\/span><span class=\"n\">ModelViewSet<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">serializer_class<\/span> <span class=\"o\">=<\/span> <span class=\"n\">PersonSerializer<\/span>\n    <span class=\"n\">queryset<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Person<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span><span class=\"o\">.<\/span><span class=\"n\">all<\/span><span class=\"p\">()<\/span>\n<\/code><\/pre><\/div>\n\n<p>And you've also got some JavaScript code that talks to this view:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\">\/\/ Inside your frontend JavaScript codebase.<\/span><span class=\"w\"><\/span>\n<span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">createPerson<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"nx\">personData<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"p\">=&gt;<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"nx\">requestData<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"nx\">method<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;POST&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"nx\">body<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"nb\">JSON<\/span><span class=\"p\">.<\/span><span class=\"nx\">stringify<\/span><span class=\"p\">(<\/span><span class=\"nx\">personData<\/span><span class=\"p\">),<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"c1\">\/\/ etc.<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"p\">}<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">response<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"k\">await<\/span><span class=\"w\"> <\/span><span class=\"nx\">fetch<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;\/api\/person\/&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"nx\">requestData<\/span><span class=\"p\">)<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"k\">return<\/span><span class=\"w\"> <\/span><span class=\"k\">await<\/span><span class=\"w\"> <\/span><span class=\"nx\">resp<\/span><span class=\"p\">.<\/span><span class=\"nx\">json<\/span><span class=\"p\">()<\/span><span class=\"w\"><\/span>\n<span class=\"p\">}<\/span><span class=\"w\"><\/span>\n<\/code><\/pre><\/div>\n\n<p>The problem occurs when you try to use the data fetched from the backend and it is using the wrong variable naming style:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\">\/\/ Inside your frontend JavaScript codebase.<\/span><span class=\"w\"><\/span>\n<span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">personData<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"nx\">full_name<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;Matt Segal&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"nx\">biggest_problem<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;My pants are too red&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"><\/span>\n<span class=\"p\">}<\/span><span class=\"w\"><\/span>\n<span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">person<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"nx\">createPerson<\/span><span class=\"p\">(<\/span><span class=\"nx\">personData<\/span><span class=\"p\">).<\/span><span class=\"nx\">then<\/span><span class=\"p\">(<\/span><span class=\"nx\">console<\/span><span class=\"p\">.<\/span><span class=\"nx\">log<\/span><span class=\"p\">)<\/span><span class=\"w\"><\/span>\n<span class=\"c1\">\/\/ {<\/span><span class=\"w\"><\/span>\n<span class=\"c1\">\/\/   full_name: &#39;Matt Segal&#39;,<\/span><span class=\"w\"><\/span>\n<span class=\"c1\">\/\/   biggest_problem: &#39;My pants are too red&#39;,<\/span><span class=\"w\"><\/span>\n<span class=\"c1\">\/\/ }<\/span><span class=\"w\"><\/span>\n<\/code><\/pre><\/div>\n\n<p>This usage of snake case in JavaScript is a little yucky and it's a quick fix.<\/p>\n<h3>The solution: install more JavaScript libraries<\/h3>\n<p>Hint: the solution is always to add more dependencies.<\/p>\n<p>To fix this we'll install <a href=\"https:\/\/www.npmjs.com\/package\/snakeize\">snakeize<\/a> and <a href=\"https:\/\/www.npmjs.com\/package\/camelize\">camelize<\/a> using npm or yarn:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>yarn add snakeize camelize\n<\/code><\/pre><\/div>\n\n<p>Then you just need to include it in your frontend's API functions:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\">\/\/ Inside your frontend JavaScript codebase.<\/span><span class=\"w\"><\/span>\n<span class=\"k\">import<\/span><span class=\"w\"> <\/span><span class=\"nx\">camelize<\/span><span class=\"w\"> <\/span><span class=\"kr\">from<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;camelize&#39;<\/span><span class=\"w\"><\/span>\n<span class=\"k\">import<\/span><span class=\"w\"> <\/span><span class=\"nx\">snakeize<\/span><span class=\"w\"> <\/span><span class=\"kr\">from<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;snakeize&#39;<\/span><span class=\"w\"><\/span>\n\n<span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">createPerson<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"nx\">personData<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"p\">=&gt;<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"nx\">requestData<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"nx\">method<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;POST&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"nx\">body<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"nb\">JSON<\/span><span class=\"p\">.<\/span><span class=\"nx\">stringify<\/span><span class=\"p\">(<\/span><span class=\"nx\">snakeize<\/span><span class=\"p\">(<\/span><span class=\"nx\">personData<\/span><span class=\"p\">)),<\/span><span class=\"w\"><\/span>\n<span class=\"w\">      <\/span><span class=\"c1\">\/\/ etc.<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"p\">}<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">response<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"k\">await<\/span><span class=\"w\"> <\/span><span class=\"nx\">fetch<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;\/api\/person\/&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"nx\">requestData<\/span><span class=\"p\">)<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">responseData<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"k\">await<\/span><span class=\"w\"> <\/span><span class=\"nx\">resp<\/span><span class=\"p\">.<\/span><span class=\"nx\">json<\/span><span class=\"p\">()<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"k\">return<\/span><span class=\"w\"> <\/span><span class=\"nx\">camelize<\/span><span class=\"p\">(<\/span><span class=\"nx\">responseData<\/span><span class=\"p\">)<\/span><span class=\"w\"><\/span>\n<span class=\"p\">}<\/span><span class=\"w\"><\/span>\n<\/code><\/pre><\/div>\n\n<p>Now we can use <code>camelCase<\/code> in the frontend and it will automatically be transformed to <code>snake_case<\/code> before it gets sent to the backend:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\">\/\/ Inside your frontend JavaScript codebase.<\/span><span class=\"w\"><\/span>\n<span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">personData<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"nx\">fullName<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;Matt Segal&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"><\/span>\n<span class=\"w\">    <\/span><span class=\"nx\">biggestProblem<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;I ate too much fish&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"><\/span>\n<span class=\"p\">}<\/span><span class=\"w\"><\/span>\n<span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">person<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"nx\">createPerson<\/span><span class=\"p\">(<\/span><span class=\"nx\">personData<\/span><span class=\"p\">).<\/span><span class=\"nx\">then<\/span><span class=\"p\">(<\/span><span class=\"nx\">console<\/span><span class=\"p\">.<\/span><span class=\"nx\">log<\/span><span class=\"p\">)<\/span><span class=\"w\"><\/span>\n<span class=\"c1\">\/\/ {<\/span><span class=\"w\"><\/span>\n<span class=\"c1\">\/\/   fullName: &#39;Matt Segal&#39;,<\/span><span class=\"w\"><\/span>\n<span class=\"c1\">\/\/   biggestProblem: &#39;I ate too much fish&#39;,<\/span><span class=\"w\"><\/span>\n<span class=\"c1\">\/\/ }<\/span><span class=\"w\"><\/span>\n<\/code><\/pre><\/div>\n\n<p>That's it! Hope this helps your eyes a little.<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"How to manage logs with Django, Gunicorn and NGINX","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/django-gunicorn-nginx-logging.html","rel":"alternate"}},"published":"2020-07-26T12:00:00+10:00","updated":"2020-07-26T12:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-07-26:\/django-gunicorn-nginx-logging.html","summary":"<p>So you want to run a Django app using NGINX and Gunicorn.\nDid you notice that <em>all three<\/em> of these tools have logging options?\nYou can configure <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/topics\/logging\/\">Django logging<\/a>, \n<a href=\"https:\/\/docs.gunicorn.org\/en\/latest\/settings.html#errorlog\">Gunicorn logging<\/a>, and <a href=\"https:\/\/docs.nginx.com\/nginx\/admin-guide\/monitoring\/logging\/\">NGINX logging<\/a>.<\/p>\n<p>You just want to see what's happening in your Django app so that you can fix \u2026<\/p>","content":"<p>So you want to run a Django app using NGINX and Gunicorn.\nDid you notice that <em>all three<\/em> of these tools have logging options?\nYou can configure <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/topics\/logging\/\">Django logging<\/a>, \n<a href=\"https:\/\/docs.gunicorn.org\/en\/latest\/settings.html#errorlog\">Gunicorn logging<\/a>, and <a href=\"https:\/\/docs.nginx.com\/nginx\/admin-guide\/monitoring\/logging\/\">NGINX logging<\/a>.<\/p>\n<p>You just want to see what's happening in your Django app so that you can fix bugs. How are you supposed to set these logs up? What are they all for?\nIn this post I'll give you a brief overview of your logging options with Django, Gunicorn and NGINX, so that you don't feel so confused and overwhelmed.<\/p>\n<p>I've previously written a short guide on <a href=\"https:\/\/mattsegal.dev\/file-logging-django.html\">setting up file logging<\/a> with Django if you just want quick instructions on what to do. <\/p>\n<h2>NGINX logging<\/h2>\n<p>NGINX allows you to set up <a href=\"https:\/\/docs.nginx.com\/nginx\/admin-guide\/monitoring\/logging\/\">two log files<\/a>, access_log and error_log. I usually configure them like this in my <code>\/etc\/nginx\/nginx.conf<\/code> file:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>access_log \/var\/log\/nginx\/access.log;\nerror_log \/var\/log\/nginx\/error.log;\n<\/code><\/pre><\/div>\n\n<h2>NGINX access logs<\/h2>\n<p>The NGINX access_log is a file which records of all the requests that are coming in to your server via NGINX. It looks like this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>123.45.67.89 - - [26\/Jul\/2020:04:55:28 +0000] &quot;GET \/ HTTP\/1.1&quot; 200 906 &quot;-&quot; &quot;Mozilla\/5.0 ... Chrome\/98 Safari\/537.4&quot;\n123.45.67.89 - - [26\/Jul\/2020:05:06:29 +0000] &quot;GET \/ HTTP\/1.1&quot; 200 904 &quot;-&quot; &quot;Mozilla\/5.0 ... Chrome\/98 Safari\/537.4&quot;\n123.45.67.89 - - [26\/Jul\/2020:05:10:33 +0000] &quot;GET \/ HTTP\/1.1&quot; 200 904 &quot;-&quot; &quot;Mozilla\/5.0 ... Chrome\/98 Safari\/537.4&quot;\n123.45.67.89 - - [26\/Jul\/2020:05:21:33 +0000] &quot;GET \/ HTTP\/1.1&quot; 200 910 &quot;-&quot; &quot;Mozilla\/5.0 ... Chrome\/98 Safari\/537.4&quot;\n123.45.67.89 - - [26\/Jul\/2020:05:25:37 +0000] &quot;GET \/ HTTP\/1.1&quot; 200 907 &quot;-&quot; &quot;Mozilla\/5.0 ... Chrome\/98 Safari\/537.4&quot;\n<\/code><\/pre><\/div>\n\n<p>There's a new line for each request that comes in. Breaking a single like down:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>123.45.67.89 - - [26\/Jul\/2020:04:55:28 +0000] &quot;GET \/ HTTP\/1.1&quot; 200 906 &quot;-&quot; &quot;Mozilla\/5.0 ... Chrome\/98 Safari\/537.4&quot;\n<\/code><\/pre><\/div>\n\n<p>From this line can see:<\/p>\n<ul>\n<li>the IP is 123.45.67.89<\/li>\n<li>the request arrived at 26\/Jul\/2020:04:55:28 +0000<\/li>\n<li>the HTTP request method was GET<\/li>\n<li>the path requested was \/<\/li>\n<li>the version of HTTP used was HTTP\/1.1<\/li>\n<li>the status code returned by the server was \"200\" (ie. <a href=\"https:\/\/http.cat\/\">OK<\/a>)<\/li>\n<li>the requester's <a href=\"https:\/\/developer.mozilla.org\/en-US\/docs\/Web\/HTTP\/Headers\/User-Agent\">user agent<\/a> was \"Mozilla\/5.0 ... Chrome\/98 Safari\/537.4\"<\/li>\n<\/ul>\n<p>This is <em>very<\/em> useful information to have when debugging issues in production, and I recommend you enable these access logs in NGINX.\nYou can quickly view these logs using <code>tail<\/code>:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># View last 5 log lines<\/span>\ntail -n <span class=\"m\">5<\/span> \/var\/log\/nginx\/access.log\n<span class=\"c1\"># View last 5 log lines and watch for new ones<\/span>\ntail -n <span class=\"m\">5<\/span> -f \/var\/log\/nginx\/access.log\n<\/code><\/pre><\/div>\n\n<p>In addition to legitimate requests to your web application, NGINX will also log all of the spam, crawlers, and hacking attempts that hit your webserver.\nIf you have your server accessible via the internet, then you will get garbage requests like this in your access log: <\/p>\n<div class=\"highlight\"><pre><span><\/span><code>195.54.160.21 - - [26\/Jul\/2020:03:58:25 +0000] &quot;POST \/vendor\/phpunit\/phpunit\/src\/Util\/PHP\/eval-stdin.php HTTP\/1.1&quot; 404 564 &quot;-&quot; &quot;Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/78.0.3904.108 Safari\/537.36&quot;\n<\/code><\/pre><\/div>\n\n<p>I assume this is a bot trying to hack an old version of PHP (which I do not run on this server).<\/p>\n<h2>NGINX error logs<\/h2>\n<p>NGINX also logs errors to error_log, which can occur when you've messed up your configuration somehow, or if your Gunicorn server is unresponsive. This file is also useful for debugging so I recommend you include it as well in your NGINX config. You get error messages like this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>2020\/07\/25 08:14:57 [error] 32115#32115: *44242 connect() failed (111: Connection refused) while connecting to upstream, client: 11.22.33.44, server: www.example.com, request: &quot;GET \/admin\/ HTTP\/1.1&quot;, upstream: &quot;http:\/\/127.0.0.1:8000\/admin\/&quot;, host: &quot;clerk.anikalegal.com&quot;, referrer: &quot;https:\/\/www.example.com\/admin\/&quot;\n<\/code><\/pre><\/div>\n\n<div class=\"ui divider\" style=\"margin: 1.5em 0;\"><\/div>\n<form action=\"https:\/\/dev.us19.list-manage.com\/subscribe\/post?u=e7a1ec466f7bb1732dbd23fc7&amp;id=ec345473bd\" method=\"post\" name=\"mc-embedded-subscribe-form\" target=\"_blank\" style=\"text-align: center; padding-bottom: 1em;\" novalidate>\n  <h3 class=\"subscribe-cta\">Get alerted when I publish new blog posts<\/h3>\n  <div class=\"ui fluid action input subscribe\">\n    <input\n      type=\"email\"\n      value=\"\"\n      name=\"EMAIL\"\n      placeholder=\"Enter your email address\"\n    \/>\n    <button class=\"ui primary button\" type=\"submit\" name=\"subscribe\">\n      Subscribe\n    <\/button>\n  <\/div>\n  <div style=\"position: absolute; left: -5000px;\" aria-hidden=\"true\">\n    <input\n      type=\"text\"\n      name=\"b_e7a1ec466f7bb1732dbd23fc7_ec345473bd\"\n      tabindex=\"-1\"\n      value=\"\"\n    \/>\n  <\/div>\n<\/form>\n<div class=\"ui divider\" style=\"margin: 1.5em 0;\"><\/div>\n\n<h2>Gunicorn logging<\/h2>\n<p>Gunicorn has <a href=\"https:\/\/docs.gunicorn.org\/en\/latest\/settings.html#errorlog\">two main logfiles<\/a> that it writes, the error log and the access log.\nYou can configure the log settings through the <a href=\"https:\/\/docs.gunicorn.org\/en\/latest\/configure.html#command-line\">command line<\/a> or a <a href=\"https:\/\/docs.gunicorn.org\/en\/latest\/configure.html#configuration-file\">config file<\/a>. I recommend using the config file because it's easier to read.<\/p>\n<h2>Gunicorn access logs<\/h2>\n<p>The Gunicorn access log is very similar to the NGINX access log, it records all the requests coming in to the Gunicorn server:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>10.255.0.2 - - [26\/Jul\/2020:05:10:33 +0000] &quot;GET \/foo\/ HTTP\/1.0&quot; 200 1938 &quot;-&quot; &quot;Mozilla\/5.0 ... (StatusCake)&quot;\n10.255.0.2 - - [26\/Jul\/2020:05:25:37 +0000] &quot;GET \/foo\/ HTTP\/1.0&quot; 200 1938 &quot;-&quot; &quot;Mozilla\/5.0 ... (StatusCake)&quot;\n10.255.0.2 - - [26\/Jul\/2020:05:40:42 +0000] &quot;GET \/foo\/ HTTP\/1.0&quot; 200 1938 &quot;-&quot; &quot;Mozilla\/5.0 ... (StatusCake)&quot;\n<\/code><\/pre><\/div>\n\n<p>I think you may as well enable this so that you can debug issues where you're not sure if NGINX is sending requests to Gunicorn properly.<\/p>\n<h2>Gunicorn error logs<\/h2>\n<p>The Gunicorn error log is a little bit more complicated. By default it contains information about what the Gunicorn server is doing, like starting up and shutting down:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>[2020-04-06 06:17:23 +0000] [53] [INFO] Starting gunicorn 20.0.4\n[2020-04-06 06:17:23 +0000] [53] [INFO] Listening at: http:\/\/0.0.0.0:8000 (53)\n[2020-04-06 06:17:23 +0000] [53] [INFO] Using worker: sync\n[2020-04-06 06:17:23 +0000] [56] [INFO] Booting worker with pid: 56\n[2020-04-06 06:17:23 +0000] [58] [INFO] Booting worker with pid: 58\n<\/code><\/pre><\/div>\n\n<p>You can change how verbose these messages are using the \"<a href=\"https:\/\/docs.gunicorn.org\/en\/latest\/settings.html#loglevel\">loglevel<\/a>\" setting, which can be set to log more info using the \"debug\" level, or only errors, using the \"error\" level, etc.<\/p>\n<p>Finally, and importantly there is the \"<a href=\"https:\/\/docs.gunicorn.org\/en\/latest\/settings.html#capture-output\">capture_output<\/a>\" logging setting, which is a boolean flag.\nThis setting will take any stdout\/stderr, which is to say print statements, log messages, warnings and errors from your Django app, and log then to the Gunicorn error file. \nI like to keep this setting enabled so that I can catch any random output that is falling through from Django to Gunicorn.\nHere is an example Gunicorn config file with logging set up:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># gunicorn.conf.py<\/span>\n<span class=\"c1\"># Non logging stuff<\/span>\n<span class=\"n\">bind<\/span> <span class=\"o\">=<\/span> <span class=\"s2\">&quot;0.0.0.0:80&quot;<\/span>\n<span class=\"n\">workers<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">3<\/span>\n<span class=\"c1\"># Access log - records incoming HTTP requests<\/span>\n<span class=\"n\">accesslog<\/span> <span class=\"o\">=<\/span> <span class=\"s2\">&quot;\/var\/log\/gunicorn.access.log&quot;<\/span>\n<span class=\"c1\"># Error log - records Gunicorn server goings-on<\/span>\n<span class=\"n\">errorlog<\/span> <span class=\"o\">=<\/span> <span class=\"s2\">&quot;\/var\/log\/gunicorn.error.log&quot;<\/span>\n<span class=\"c1\"># Whether to send Django output to the error log <\/span>\n<span class=\"n\">capture_output<\/span> <span class=\"o\">=<\/span> <span class=\"kc\">True<\/span>\n<span class=\"c1\"># How verbose the Gunicorn error logs should be <\/span>\n<span class=\"n\">loglevel<\/span> <span class=\"o\">=<\/span> <span class=\"s2\">&quot;info&quot;<\/span>\n<\/code><\/pre><\/div>\n\n<p>You can run Gunicorn using config like this as follows:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>gunicorn myapp.wsgi:application -c \/some\/folder\/gunicorn.conf.py\n<\/code><\/pre><\/div>\n\n<h2>Django logging<\/h2>\n<p>Django logging refers to the output of your Django application. The kind of messages you see printed by <code>runserver<\/code> in development. Stuff like this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>Sending Thing&lt;b5d1854b-7efc-4c67-9e9b-a956c10e5b86]&gt; to Google API\nGoogle API called failed: {&#39;error_description&#39;: &#39;You failed hahaha&#39;}\nTraceback (most recent call last):\n  File &quot;\/app\/google\/api\/base.py&quot;, line 102, in _handle_json_response\n    resp.raise_for_status()\n  File &quot;\/usr\/local\/lib\/python3.6\/dist-packages\/requests\/models.py&quot;\n    raise HTTPError(http_error_msg, response=self)\nrequests.exceptions.HTTPError: 403 Client Error\nSetting expired tokens to inactive: []\n<\/code><\/pre><\/div>\n\n<p>I discuss Django logging in more detail in <a href=\"https:\/\/mattsegal.dev\/file-logging-django.html\">this guide<\/a>, but I will give you a brief summary here.\nDjango uses the same conventions as Python's standard library <a href=\"https:\/\/docs.python.org\/3\/library\/logging.html\">logging<\/a> module, which is kind of a pain to learn, but valuable to know.\nThe Django docs provide a nice overview of logging config <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/topics\/logging\/\">here<\/a>.<\/p>\n<p>I think you have two viable options for your Django logging:<\/p>\n<ul>\n<li>Set up Django to log everything to stdout\/stderr using the <code>StreamHandler<\/code> and capture the output using Gunicorn via the capture_output option, so that your Django logs end up in the Gunicorn error logfile<\/li>\n<li>Set up Django to log to a file using <code>FileHandler<\/code> so you can keep your Django and Gunicorn logs separate<\/li>\n<\/ul>\n<p>I personally prefer option #2, but you whatever makes you happy.<\/p>\n<h2>Next steps<\/h2>\n<p>I encourage you to set up the logging described in this post, so that you don't waste hours trying to figure out what is causing bugs in production.\nI also recommend that you configure error alerting with Django, with <a href=\"https:\/\/mattsegal.dev\/sentry-for-django-error-monitoring.html\">Sentry<\/a> being a strong choice.<\/p>\n<p>Finally, if you're having other difficulties getting your Django app onto the internet, then check out my guide on <a href=\"https:\/\/mattsegal.dev\/simple-django-deployment.html\">Django deployment<\/a><\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"How to make your Django project easy to move and share","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/django-portable-setup.html","rel":"alternate"}},"published":"2020-07-24T12:00:00+10:00","updated":"2020-07-24T12:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-07-24:\/django-portable-setup.html","summary":"<p>You need your Django project to be portable. It should be quick and easy to start it up on a new laptop. \nIf it isn't portable, then your project is trapped on your machine. If it gets deleted or corrupted, then you've lost all your work!\nThis issue comes up \u2026<\/p>","content":"<p>You need your Django project to be portable. It should be quick and easy to start it up on a new laptop. \nIf it isn't portable, then your project is trapped on your machine. If it gets deleted or corrupted, then you've lost all your work!\nThis issue comes up in quite a few scenarios:<\/p>\n<ul>\n<li>you want to work on your code on multiple machines, like a laptop and a PC<\/li>\n<li>you want to get help from other people, and they want to try running your code<\/li>\n<li>you somehow screwed up your files very badly and you want to start from scratch <\/li>\n<\/ul>\n<p>In the worst case, moving your Django project from one machine to another is a frustrating and tedious experience that involves dead ends, mystery bugs and cryptic error messages. It's the kind of thing that makes you want to scream at your computer.<\/p>\n<p><img alt=\"frutsrated fox\" src=\"https:\/\/mattsegal.dev\/img\/frustrated-fox.jpeg\"><\/p>\n<p>In the best case, this process can take minutes. To achieve this best case, there are some steps that you'll\nneed to take to make your development environment reproducable.<\/p>\n<p>If you don't believe that this is achievable, then here's a quick example of me cloning and setting up an <a href=\"git@github.com:MattSegal\/djdt-perf-demo.git\">example project<\/a> from scratch in under a minute:<\/p>\n<div class=\"loom-embed\"><iframe src=\"https:\/\/www.loom.com\/embed\/01cbd6d2c2f04d0ab78e4d33d0174de5\" frameborder=\"0\" webkitallowfullscreen mozallowfullscreen allowfullscreen style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%;\"><\/iframe><\/div>\n\n<p>In the rest of this post, I'll describe some practices that will help ensure that anyone with Python installed can quickly start working on your Django app.<\/p>\n<h2>Hosting your code<\/h2>\n<p>The best way to make your code portable between multiple computers is to put it online in a place that is publicly accessible, like <a href=\"https:\/\/github.com\/\">GitHub<\/a>.\nFor example, this blog is <a href=\"https:\/\/github.com\/MattSegal\/devblog\">hosted on GitHub<\/a> so that I can access the latest copy of my writing from both my laptop and PC.\nGit, the version control tool, is widely used by software developers and allows you to efficently and reliably sync your code between multiple machines.<\/p>\n<p>If you don't know Git and you plan to work with software in any capacity, then I strongly recommend that you start learning how to use it as soon as possible.\nThere are plenty of <a href=\"https:\/\/hellowebbooks.com\/learn-git\/\">books<\/a>, <a href=\"https:\/\/www.udacity.com\/course\/version-control-with-git--ud123\">online courses<\/a>, <a href=\"https:\/\/missing.csail.mit.edu\/2020\/version-control\/\">lectures<\/a> and <a href=\"https:\/\/try.github.io\/\">more<\/a> to help you learn. It's a pain in the ass to start with, no doubt about that, but it is definitely worth your time.<\/p>\n<h2>Tracking Python dependencies<\/h2>\n<p>Your project needs a bunch of 3rd party libraries to run. Obviously Django is required, plus maybe, Django Rest Framework, Boto3... Pillow, perhaps?\nIt's hard to remember all the thing that you've <code>pip install<\/code>'d, which is why it's really important to track all the libraries that your app needs, plus the versions, if those are important to you.<\/p>\n<p>There is a Python convention of tracking all your libraries in a <a href=\"https:\/\/pip.pypa.io\/en\/latest\/reference\/pip_install\/#example-requirements-file\">requirements.txt<\/a> file.\nExperienced Python devs immediately know what to do if they see a project with one of these files, so it's good if you stick with this practice. Installing all\nyour requirements is as easy as:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>pip install -r requirements.txt\n<\/code><\/pre><\/div>\n\n<p>You can also use <code>pip freeze<\/code> to get an exact snapshot of your current Python packages and write them to a file:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>pip freeze &gt; requirements.txt\n<\/code><\/pre><\/div>\n\n<p>Python's <code>pip<\/code> package manager tries to install all of your dependencies in your global system Python folder by default, which is a really dumb idea, \nand it can cause issues where multiple Python projects are all installing libraries in the same place. When this happens you can get the wrong\nversion installed, and you can no longer keep track of what dependencies you need to run your code, because they're are muddled\ntogether with the ones from all your other projects.<\/p>\n<p>The simplest way to fix this issue is to <em>always<\/em> use <code>virtualenv<\/code> to isolate your Python dependencies. You can read a guide on that <a href=\"https:\/\/realpython.com\/python-virtual-environments-a-primer\/\">here<\/a>.  Using <code>virtualenv<\/code>, incidentally, also fixes the problem where you sometimes have to use <code>sudo<\/code> to pip install things on Linux.\nThere are also other tools like <a href=\"https:\/\/realpython.com\/pipenv-guide\/\">pipenv<\/a> or <a href=\"https:\/\/python-poetry.org\/\">poetry<\/a> that solve this problem as well. Use whatever you want,\nbut it's a good idea to pick <em>something<\/em>, or you will shed many tears over Python dependency errors in the future.<\/p>\n<h2>Repeatable setup instructions<\/h2>\n<p>Most simple Django projects have the exact same setup sequence. It's almost always roughly this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># Create and activate virtual environment<\/span>\nvirtualenv -p python3 env\n. .\/env\/bin\/activate\n\n<span class=\"c1\"># Install Python dependencies<\/span>\npip install -r requirements.txt\n\n<span class=\"c1\"># Create SQLite databse, run migrations<\/span>\n<span class=\"nb\">cd<\/span> myapp\n.\/manage.py migrate\n\n<span class=\"c1\"># Run Django dev server<\/span>\n.\/manage.py runserver\n<\/code><\/pre><\/div>\n\n<p>But for anything but the simplest projects there's usually a few extra steps that you'll need to get up and running.\nYou need to <strong>write this shit down<\/strong>, preferably in your project's README, or <strong>you will forget<\/strong>.\nEven if you remember all these steps, your friends or colleagues will get stuck if they're not available.<\/p>\n<p>You want to document all the instructions that someone needs to do to start running your project, with as much of it being explicit\nline of shell code as possible. Someone, who already has Python setup, should be able to clone your project onto their laptop with Git, follow\nyour instructions, and then be able to run your Django app. The kind of extra things that you should document are:<\/p>\n<ul>\n<li>any extra scripts or management commands that the user must run<\/li>\n<li>any environment variables or files that the user needs to configure<\/li>\n<li>setup of required data in the Django admin or shell <\/li>\n<li>installing and running any 3rd party dependencies (eg. Docker, Postgres, Redis)<\/li>\n<li>building required front end web assets (eg. with Webpack)<\/li>\n<li>downloading essential data from the internet<\/li>\n<\/ul>\n<p>Documenting the project setup isn't so important for small and simple projects, but it's also really easy to do (see script above). \nAs your project becomes more complicated, the need to have replicable, explicit setup instructions becomes vital. \nIf you do not maintain these instructions, then it will cost your hours of work when you forget to perform a vital step and your app doesn't work.<\/p>\n<p>I've written before on <a href=\"https:\/\/mattsegal.dev\/github-resume-polish.html#readme\">how to write a nice README<\/a>, which you might find useful.\nIt's a little over the top for the purposes of just making your project portable and reproducible, but it should give you a general idea of what to cover.<\/p>\n<h2>Exclude unnecessary files<\/h2>\n<p>Your project should only contain source code, plus the minimum files required to run it. It should not not contain:<\/p>\n<ul>\n<li>Editor config files (.idea, .vscode)<\/li>\n<li>Database files (eg. SQLite)<\/li>\n<li>Random documents (.pdf, .xls)<\/li>\n<li>Non-essential media files (images, videos, audio)<\/li>\n<li>Bytecode (eg. *.pyc files)<\/li>\n<li>Build artifacts (eg. JavaScript and CSS from Webpack)<\/li>\n<li>Virtual environments (eg env\/venv folders)<\/li>\n<li>JavaScript packages (node_modules)<\/li>\n<li>Log files (eg. *.log)<\/li>\n<\/ul>\n<p>Some of these files are just clutter, but the SQLite databases and bytecode are particularly important to exclude.<\/p>\n<p>SQLite files are a binary format, which Git does not store easily. Every change to the database causes Git to store a whole new copy.\nIn addition, there's no way to \"merge\" databases with Git, meaning the data will get regularly overwritten by multiple users.<\/p>\n<p><a href=\"https:\/\/opensource.com\/article\/18\/4\/introduction-python-bytecode\">Python bytecode<\/a> files, with the <code>.pyc<\/code> extension, can cause issues\nwhen shared between different machines, and are also just yucky to look at.<\/p>\n<p>You can exlude all of the files (and folders) I described above using a <code>.gitignore<\/code> file, in the root of your repository, with contents something like this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code># General\n*.log\n*.pdf\n*.png\n\n# IDE\n.idea\/ # PyCharm settings\n.vscode\/ # VSCode settings\n\n# Python\n*.pyc\nenv\/\nvenv\/\n\n# Databases\n*.sqlite3\n\n# JavaScript\nnode_modules\/\nbuild\/ # Webpack build output\n<\/code><\/pre><\/div>\n\n<p>If you've already added these kinds of files to your project's Git history, then you'll need to delete them before ignoring them.<\/p>\n<p>In addition, a common mistake by beginners is to exclude migration files from theit Git history. Django migration files belong in source control,\nso that you can ensure that everybody is running the same migrations on their data.<\/p>\n<h2>Automate common tasks<\/h2>\n<p>Although it's not strictly necessary, it's really nice to automate your project setup, so that you can get started by just running a few scripts.\nYou can use bash scripts if you're a Linux or Mac user, PowerShell if you're using Windows, or even custom Django management commands. I also recommend checking out <a href=\"https:\/\/www.pyinvoke.org\/\">Invoke<\/a>, which is a nice, cross-platform Python tool for running tasks (<a href=\"https:\/\/github.com\/MattSegal\/link-sharing-site\/blob\/master\/tasks.py\">example Invoke script<\/a>).<\/p>\n<p>For example, in this <a href=\"https:\/\/github.com\/MattSegal\/djdt-perf-demo\">demo repo<\/a>, I added a script which <a href=\"https:\/\/mattsegal.dev\/django-factoryboy-dummy-data.html\">fills the website with test data<\/a>, which a user can quickly run via a management command:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>.\/manage.py setup_test_data\n<\/code><\/pre><\/div>\n\n<p>In other projects of mine, I also like to include a script that allows me to <a href=\"https:\/\/mattsegal.dev\/restore-django-local-database.html\">pull production data into my local database<\/a>, which is also just one quick copy-paste to run. <\/p>\n<div class=\"highlight\"><pre><span><\/span><code>.\/scripts\/restore-prod.sh\n<\/code><\/pre><\/div>\n\n<h2>Next steps<\/h2>\n<p>If you're working on a Django project right now, I recommend that you make sure that it's portable. \nIt doesn't take long to do and you will save yourself hours and hours of this:<\/p>\n<p><img alt=\"dog screaming internally\" src=\"https:\/\/mattsegal.dev\/img\/screams.jpg\"><\/p>\n<p>If multiple people are working on your Django project and you want to become even more productive as a team, then I also recommend that you begin <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/topics\/testing\/\">writing tests<\/a> and <a href=\"https:\/\/mattsegal.dev\/pytest-on-github-actions.html\">run them automatically with GitHub Actions<\/a>.<\/p>\n<p>If you've found moving your Django project around to be a frustrating experience, then you've probably also had trouble deploying it to the web as well. If that's the case, you might enjoy my guide on <a href=\"https:\/\/mattsegal.dev\/simple-django-deployment.html\">Django deployment<\/a>, where I show you how to deploy Django to a DigitalOcean virtual machine.<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"Is Django too slow?","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/is-django-too-slow.html","rel":"alternate"}},"published":"2020-07-24T12:00:00+10:00","updated":"2020-07-24T12:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-07-24:\/is-django-too-slow.html","summary":"<p>Does Django have \"bad performance\"?\nThe framework is now 15 years old. Is it out of date?\nMostly, no. I think that Django's performance is perfectly fine for most use-cases.\nIn this post I'll review different aspects of Django's \"performance\" as a web framework and discuss how you can decide \u2026<\/p>","content":"<p>Does Django have \"bad performance\"?\nThe framework is now 15 years old. Is it out of date?\nMostly, no. I think that Django's performance is perfectly fine for most use-cases.\nIn this post I'll review different aspects of Django's \"performance\" as a web framework and discuss how you can decide whether it's a good fit for your web app.<\/p>\n<h2>Benchmarks<\/h2>\n<p>Let's start by digging into the ad-hoc web app performance benchmarks that you'll see pop up on Medium from time to time. To produce a graph like the one below, the author of <a href=\"https:\/\/medium.com\/@mihaigeorge.c\/web-rest-api-benchmark-on-a-real-life-application-ebb743a5d7a3\">this article<\/a> sets up a server for each of the frameworks tested and sends them a bunch of HTTP requests. The benchmarking tool counts number of requests served per second by each framework.<\/p>\n<p><img alt=\"benchmark\" src=\"https:\/\/mattsegal.dev\/img\/benchmark.png\"><\/p>\n<p>I think these kind of measurements are irrelevant to practical web development. There are a few factors to consider:<\/p>\n<ul>\n<li>Is the metric being measured actually of interest? What's a good baseline? Is 100 requests per seconds good, or pathetic? Is 3000 requests\/s practically better than 600 requests\/s?<\/li>\n<li>Is the test representative of an actual web app workload? In this case, how often do we just send a static \"hello world\" JSON to users?<\/li>\n<li>Are we comparing apples to apples? For example, ExpressJS has 3 layers of relatively simple middleware enabled by default, wheras Django provides a larger stack of middleware features, \"out of the box\"<\/li>\n<li>Has each technology been set up correctly? Was Gunicorn, for example, run with an optimal number of workers?<\/li>\n<\/ul>\n<p>This kind of naive comparsison is a little misleading and it's hard to use it to make practical decisions. So, what kind of performance metrics should you pay attention to when working on your Django app?<\/p>\n<h2>What do you mean by \"performance\"?<\/h2>\n<p>When you ask whether a framework or language is \"slow\", you should also ask \"slow at what?\" and \"why do you care?\".\nFundamentally I think there are really only two performance goals: a good user experience and low hosting cost. How much money does running this website cost me, and do people enjoy using my website? For user experience I'm going to talk about two factors:<\/p>\n<ul>\n<li>Response time: how long people need to wait before their requests are fulfilled<\/li>\n<li>Concurrency: how many people can use your website at the same time<\/li>\n<\/ul>\n<p>Cost, on the other hand, is typically proportional to compute resources: how many CPU cores and GB of RAM you will need to run your web app.<\/p>\n<h2>Response time in Django<\/h2>\n<p>Users don't like waiting for their page to load, so the less time they have to wait, the better. There are a few different\nmetrics that you could use to measure page load speed, such as <a href=\"https:\/\/web.dev\/time-to-first-byte\/\">time to first byte<\/a> or <a href=\"https:\/\/web.dev\/first-contentful-paint\/\">first contentful paint<\/a>, both of which you can check with <a href=\"https:\/\/developers.google.com\/speed\/pagespeed\/insights\/\">PageSpeed Insights<\/a>. Faster responses don't benefit your user linearly though, not every 5x improvement in response is equally beneficial. A user getting a response in:<\/p>\n<ul>\n<li>5s compared to 25s transforms the app from \"broken\" to \"barely useable\"<\/li>\n<li>1s compared to 5s is a huge improvement<\/li>\n<li>200ms instead of 1s is good<\/li>\n<li>50ms instead of 200ms is nice, I guess, but many people wouldn't notice<\/li>\n<li>10ms instead of 50ms is imperceptible, no one can tell the difference<\/li>\n<\/ul>\n<p>So if someone says \"this framework is 5x faster than that framework blah blah blah\" it really doesn't mean anything without more context.\nThe important question is: will your users notice? Will they care?<\/p>\n<p>So, what makes a page load slowly in Django? The most common beginner mistakes are using too many database queries or making slow API calls to external services.\nI've written previously on how to <a href=\"https:\/\/mattsegal.dev\/django-debug-toolbar-performance.html\">find and fix slow database queries with Django Debug Toolbar<\/a> and how to <a href=\"https:\/\/mattsegal.dev\/offline-tasks.html\">push slow API calls into offline tasks<\/a>. There are <strong>many<\/strong> other ways to make your Django web pages or API endpoints load slowly, but if you avoid these two major pitfalls then you should be able to serve users with a time to first byte (TTFB) of 1000ms or less and provide a reasonable user experience.<\/p>\n<h2>When is Django's response time not fast enough?<\/h2>\n<p>Django isn't perfect for every use case, and sometimes it can't respond to queries fast enough.\nThere are some aspects of Django that are hard to optimise without giving up much of the convenience that makes the framework attractive in the first place.\nYou will always have to wait for Django when it is:<\/p>\n<ul>\n<li>running requests through middleware (on the way in and out) <\/li>\n<li>serializing and deserializing JSON strings<\/li>\n<li>building HTML strings from templates<\/li>\n<li>converting database queries into Python objects<\/li>\n<li>running garbage collection<\/li>\n<\/ul>\n<p>All this stuff run really fast on modern computers, but it is still overhead.\nMost humans don't mind waiting roughly a second for their web page to load, but machines can be more impatient.\nIf you are using Django to serve an API, where it is primarily computer programs talking to other computer programs, then it <em>may<\/em> not be fast enough for very high performance workloads. Some applications where you would consider ditching Django to shave off some latency are:<\/p>\n<ul>\n<li>a stock trading marketplace<\/li>\n<li>an global online advertisement serving network<\/li>\n<li>a low level infrastructure control API<\/li>\n<\/ul>\n<p>If you find yourself sweating about an extra 100ms here or there, then maybe it's time to look at alternative web frameworks or languages. If the difference between a 600ms and 500ms TTFB doesn't mean much to you, then Django is totally fine.<\/p>\n<h2>Concurrency in Django<\/h2>\n<p>As we saw in the benchmark above, Django web apps can handle multiple requests at the same time. This is important if your application has multiple users. If too many people try to use your site at the same time, then it will eventually become overwhelmed, and they will be served errors or timeouts. In Australia, our government's household census website was <a href=\"https:\/\/www.abc.net.au\/news\/2016-08-09\/abs-website-inaccessible-on-census-night\/7711652\">famously overwhelmed<\/a> when the entire country tried to access an online form in 2016. This effect is often called the \"<a href=\"https:\/\/en.wikipedia.org\/wiki\/Slashdot_effect\">hug of death<\/a>\" and associated with small sites becoming popular on Reddit or Hacker News.<\/p>\n<p>A Django app's <a href=\"https:\/\/mattsegal.dev\/simple-django-deployment-2.html#wsgi\">WSGI server<\/a> is the thing that handles multiple concurrent requests. I'm going to use <a href=\"https:\/\/gunicorn.org\/\">Gunicorn<\/a>, the WGSI server I know best, as a reference. Gunicorn can provide two kinds of concurrency: multiple child worker processes and multiple green threads per worker. If you don't know what a \"process\" or a \"green thread\" is then, whatever, suffice to say that you can set Gunicorn up to handle multiple requests at the same time. <\/p>\n<p>What happens if a new request comes in and all the workers\/threads are busy? I'm a little fuzzy on this, but I believe these extra requests get put in a queue, which is managed by Gunicorn. It appears that the <a href=\"https:\/\/docs.gunicorn.org\/en\/stable\/settings.html#backlog\">default length<\/a> of this queue is 2048 requests. So if the workers get overwhelmed, then the extra requests get put on the queue so that the workers can (hopefully) process them later. Typically NGINX will timeout any connections that have not received a response in 60s or less, so if a request gets put in the queue and doesn't get responded to in 60s, then the user will get a HTTP 504 \"Gateway Timeout\" error. If the queue gets full, then Gunicorn will start sending back errors for any overflowing requests.<\/p>\n<p>It's interesting to note the relationship between request throughput and response time. If your WSGI server has 10 workers\nand each request takes 1000ms to complete, then you can only serve ~10 requests per second. If you optimise your Django code so that each request only takes\n100ms to complete, then you can serve ~100 requests per second. Given this relationship, it's sometimes good to improve your app's response time even if users won't notice, because it will also improve the number of requests\/second that you can serve.<\/p>\n<p>There are some limitations to adding more Gunicorn workers, of course:<\/p>\n<ul>\n<li>Each additional worker eats up some RAM (which can be reduced if you use <a href=\"https:\/\/docs.gunicorn.org\/en\/latest\/settings.html#preload-app\">preload<\/a>)<\/li>\n<li>Each additional worker\/thread will eat some CPU when processing requests<\/li>\n<li>Each additional worker\/thread will eat some extra CPU when listening to new requests, ie. the \"<a href=\"https:\/\/docs.gunicorn.org\/en\/latest\/faq.html#does-gunicorn-suffer-from-the-thundering-herd-problem\">thundering herd problem<\/a>\", which is described in great detail <a href=\"https:\/\/rachelbythebay.com\/w\/2020\/03\/07\/costly\/\">here<\/a><\/li>\n<\/ul>\n<p>So, really, the question of \"how much concurrency can Django handle\" is actually a question of \"how much cloud compute can you afford\":<\/p>\n<ul>\n<li>if you need to handle more requests, add more workers<\/li>\n<li>if you need more RAM, rent a virtual machine with more RAM<\/li>\n<li>if you have too many workers one server and are seeing \"thundering herd\" problems, then <a href=\"https:\/\/mattsegal.dev\/django-prod-architecture\/nginx-2-external.png\">scale out your web servers<\/a> (<a href=\"https:\/\/mattsegal.dev\/django-prod-architectures.html\">more here<\/a>)<\/li>\n<\/ul>\n<p>This situation is, admittedly, not ideal, and it would be better if Gunicorn were more resource efficient. To be fair, though, this problem of scaling\nDjango's concurrency doesn't really come up for most developers. If you're working at <a href=\"https:\/\/instagram-engineering.com\/\">Instagram<\/a> or <a href=\"https:\/\/www.eventbrite.com\/engineering\/our-strategy-to-migrate-to-django\/\">Eventbrite<\/a>, then sure, this is costing your company some serious money, but most developers don't run apps that operate at a scale where this is an issue.<\/p>\n<p>How do you know if you can support enough concurrency with your current infrastructure? I recommend using <a href=\"https:\/\/locust.io\/\">Locust<\/a> to load test your app\nwith dozens, hundreds, or thousands of simultaneous users - whatever you think a realistic \"bad case\" scenario would look like. Ideally you would do this on a staging server that has a similar architecture and compute resources to your production enviroment. If your server becomes overwhelmed with requests and starts returning\nerrors or timeouts, then you know you have concurrency issues. If all requests are gracefully served, then you're OK!<\/p>\n<p>What if the traffic to your site is very \"bursty\" though, with large transient peaks, or you're afraid that you'll get the dreaded \"hug of death\"? \nIn that case I recommend looking into \"<a href=\"https:\/\/en.wikipedia.org\/wiki\/Autoscaling\">autoscaling<\/a>\" your servers, based on a metric like CPU usage.<\/p>\n<p>If you're interested, you can read more on Gunicorn <a href=\"https:\/\/docs.gunicorn.org\/en\/latest\/design.html#how-many-workers\">worker selection<\/a> and how to configure Gunicorn to <a href=\"https:\/\/medium.com\/building-the-system\/gunicorn-3-means-of-concurrency-efbb547674b7\">use more workers\/threads<\/a>. There's also <a href=\"https:\/\/medium.com\/@bfirsh\/squeezing-every-drop-of-performance-out-of-a-django-app-on-heroku-4b5b1e5a3d44\">this interesting case study<\/a> on optimising Gunicorn for <a href=\"https:\/\/www.arxiv-vanity.com\/\">arxiv-vanity.com<\/a>.<\/p>\n<h2>When is Django's concurrency not enough?<\/h2>\n<p>You will have hit the wall when you run out of money, or you can't move your app to a bigger server, or distribute it across more servers.\nIf you've twiddled all the available settings and still can't get your app to handle all the incoming requests without sending back errors or \nburning through giant piles of cash, then maybe Django isn't the right backend framework for your application.<\/p>\n<h2>The other kind of \"performance\"<\/h2>\n<p>There's one more aspect of performance to consider: your performance as a developer. Call it your <a href=\"https:\/\/en.wikipedia.org\/wiki\/Takt_time\">takt time<\/a>, if you like metrics. Your ability to quickly and easily fix bugs and ship new features is valuable to both you and your users.\nImprovements to the speed or throughput of your web app that also makes your code harder to work with may not be worth it.\nCost savings on infrastructure might be a waste if the change makes you less productive and costs you your time.<\/p>\n<p>Choosing languages, frameworks and optimisations is an engineering decision, and in all engineering decisions there are competing tradeoffs to be considered, at least at the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Pareto_efficiency\">Pareto frontier<\/a>.<\/p>\n<p>If raw performance was all we cared about, then we'd just write all our web apps in assembly.<\/p>\n<p><img alt=\"web development in assembly\" src=\"https:\/\/mattsegal.dev\/img\/assembly.webp\"><\/p>\n<h2>Next steps<\/h2>\n<p>If you liked reading about running Django in production, then you might also enjoy another post I wrote, which gives you a tour of some common <a href=\"https:\/\/mattsegal.dev\/django-prod-architectures.html\">Django production architectures<\/a>. If you've written a Django app and you're looking to deploy it to production, then\nyou might enjoy my guide on <a href=\"https:\/\/mattsegal.dev\/simple-django-deployment.html\">Django deployment<\/a>.<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"How to find what you want in the Django documentation","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/how-to-read-django-docs.html","rel":"alternate"}},"published":"2020-06-26T12:00:00+10:00","updated":"2020-06-26T12:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-06-26:\/how-to-read-django-docs.html","summary":"<p>Many beginner programmers find the <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/\">Django documentation<\/a> overwhelming.<\/p>\n<p>Let's say you want to learn how to perform a login for a user. Seems like it would be pretty simple: logins are a core feature of Django. If you <a href=\"https:\/\/www.google.com\/search?q=django+login\">google for \"django login\"<\/a> or <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/search\/?q=login\">search the docs<\/a> you see a few \u2026<\/p>","content":"<p>Many beginner programmers find the <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/\">Django documentation<\/a> overwhelming.<\/p>\n<p>Let's say you want to learn how to perform a login for a user. Seems like it would be pretty simple: logins are a core feature of Django. If you <a href=\"https:\/\/www.google.com\/search?q=django+login\">google for \"django login\"<\/a> or <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/search\/?q=login\">search the docs<\/a> you see a few options, with \"Using the Django authentication system\" as the most promising result. You click the link, happily anticipating that your login problems will soon be over, and you get smacked in the face with <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/topics\/auth\/default\/\">thirty nine full browser pages of text<\/a>. This is way too much information!<\/p>\n<p>Alternatively, you find your way to the reference page on <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/ref\/contrib\/auth\/\">django.contrib.auth<\/a>, because that's where all the auth stuff is, right? If you browse this page you will see an endless enumeration of all the different authentication models and fields and functions, but no explanation of how they're supposed to fit together.<\/p>\n<p>At this stage you may want to close your browser tab in despair and reconsider your decision to learn Django. It turns out the info that you wanted was somewhere in that really long page <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/topics\/auth\/default\/#how-to-log-a-user-in\">here<\/a> and <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/topics\/auth\/default\/#django.contrib.auth.authenticate\">here<\/a>. Why was it so hard to find? Why is this documentation so fragmented?<\/p>\n<p>God forbid that you should complain to anyone about this struggle. Experienced devs will say things like \"you are looking in the wrong place\" and \"you need more experience before you try Django\". This response begs the question though: how does anyone know where the \"right place\" is? The table of contents in the Django documentation <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/contents\/\">is unreadably long<\/a>. Meanwhile, you read other people raving about how great Django docs are: what are they talking about? You may wonder: am I missing something?<\/p>\n<p>Wouldn't it be great if you could go from having a question to finding the answer in a few minutes or less? A quick Google and a scan, and boom: you know how to solve your Django problem. This is possible. As a professional Django dev I do this daily. I rarely remember how to do anything from heart and I am constantly scanning the docs to figure out how to solve problems, and you can too.<\/p>\n<p>In this post I will outline how to find what you want in the Django documentation, so that you spend less time frustrated and stuck, and more time writing your web app. I also include a list of key references that I find useful.<\/p>\n<p>Experienced devs can be dismissive when you complain about documentation, but they're right about one thing: knowing how to read docs is a really important skill for a programmer, and being good at this will save you lots of time.<\/p>\n<h2>Find the right section<\/h2>\n<p>Library documentation is almost always written with distinct sections. If you do not understand what these sections are for, then you will be totally lost.\nIf you have time, watch <a href=\"https:\/\/www.youtube.com\/watch?v=t4vKPhjcMZg\">Daniele Procida's excellent talk<\/a> how documentation should be structured. In the talk he describes four different sections of documentation:<\/p>\n<ul>\n<li><strong>Tutorials<\/strong>: lessons that show you how to complete a small project (<a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/intro\/install\/\">example<\/a>)<\/li>\n<li><strong>How-to guides<\/strong>: guide with steps on how to solve a common problem (<a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/howto\/custom-management-commands\/\">example<\/a>)<\/li>\n<li><strong>API References<\/strong>: detailed technical descriptions of all the bits of code (<a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/ref\/models\/querysets\/\">example<\/a>)<\/li>\n<li><strong>Explanations<\/strong>: high level discussion of design decisions (<a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/topics\/templates\/#module-django.template\">example<\/a>)<\/li>\n<\/ul>\n<p>In addition to these, there's also commonly a <strong>Quickstart<\/strong> (<a href=\"http:\/\/whitenoise.evans.io\/en\/stable\/#quickstart-for-django-apps\">example<\/a>), which is the absolute minimum steps you need to to do get started with the library.<\/p>\n<p>The Django Rest Framework docs use a structure similar to this<\/p>\n<p><img alt=\"django rest framework sections\" src=\"https:\/\/mattsegal.dev\/img\/drf-sections.png\"><\/p>\n<p>The ReactJS docs use a structure similar to this<\/p>\n<p><img alt=\"react sections\" src=\"https:\/\/mattsegal.dev\/img\/react-sections.png\"><\/p>\n<p>The Django docs use a <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/#how-the-documentation-is-organized\">structure similar to this<\/a><\/p>\n<p><img alt=\"django sections\" src=\"https:\/\/mattsegal.dev\/img\/django-sections.png\"><\/p>\n<p>Hopefully you see the pattern here: all these docs have been split up into distinct sections. Learn this structure once and you can quickly navigate most documentation.\nNow that you understand that library documentation is usually structured in a particular way, I will explain how to navigate that structure.<\/p>\n<h2>Do the tutorial first<\/h2>\n<p>This might seem obvious, but I have to say it. If there is a tutorial in the docs and you are feeling lost, then do the tutorial. It is a place where the authors may have decided to introduce concepts that are key to understanding everything else. If you're feeling like a badass, then don't \"do\" the tutorial, but at the very least skim read it.<\/p>\n<h2>Find an example, guide or overview<\/h2>\n<p>Avoid the <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/ref\/\">API reference<\/a> section, unless you already know <em>exactly<\/em> what you're looking for. You will recognise that you are in an API reference section because the title will have \"reference\" in it, and the content will be very detailed with few high-level explanations. For example, <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/ref\/contrib\/auth\/\">django.contrib.auth<\/a> is a reference section - it is not a good place to learn how \"Django login\" works.<\/p>\n<p>You need to understand how the bits of code fit together before looking at an API reference. This can be hard since most documentation, even the really good stuff, is incomplete. Still, the best thing to try is to look for overviews and explanations of framework features.<\/p>\n<p>Find and scan the list of <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/howto\/\">how-to guides<\/a>, to see if they solve your exact problem. This will save you a lot of time if the guide directly solves your problem. Using our login example, there is no \"how to log a user in\" guide, which is bad luck.<\/p>\n<p>If there is no guide, then quickly scan the <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/topics\/\">topic list<\/a> and try and find the topic that you need. If you do not already understand the topic well, then read the overview. <strong>Google terms that you do not understand<\/strong>, like \"authentication\" and \"authorization\" (they're different, specific things). In our login case, \"<a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/topics\/auth\/\">User authentication in Django<\/a>\" is the topic that we want from the list.<\/p>\n<p>Once you think you sort-of understand how everything should fit together, then you can move to the detailed API reference, so that you can ensure that you're using the code correctly.<\/p>\n<h2>Find and remember key references<\/h2>\n<p>Once you understand what you want to do, you will need to use the API reference pages to figure out exactly what code you should write. It's good to remember key pages that contain the most useful references. Here's my personal favourites that I use all the time:<\/p>\n<ul>\n<li><a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/ref\/settings\/\"><strong>Settings reference<\/strong><\/a>: A list of all the settings and what they do<\/li>\n<li><a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/ref\/templates\/builtins\/\"><strong>Built-in template tags<\/strong><\/a>: All the template tags with examples<\/li>\n<li><a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/ref\/models\/querysets\/\"><strong>Queryset API reference<\/strong><\/a>: All the different tools for using the ORM to access the database<\/li>\n<li><a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/ref\/models\/fields\/\"><strong>Model field reference<\/strong><\/a>: All the different model fields<\/li>\n<li><a href=\"https:\/\/ccbv.co.uk\/\"><strong>Classy Class Based Views<\/strong><\/a>: Detailed descriptions for each of Django's class-based views<\/li>\n<\/ul>\n<p>I don't have any of these pages bookmarked, I just google for them and then search using <code>ctrl-f<\/code> to find what I need in seconds.<\/p>\n<p>When using Django REST Framework I often find myself referring to:<\/p>\n<ul>\n<li><a href=\"http:\/\/www.cdrf.co\/\"><strong>Classy DRF<\/strong><\/a>: Like Classy Class Based Views but for DRF<\/li>\n<li><a href=\"https:\/\/www.django-rest-framework.org\/api-guide\/serializers\/\"><strong>Serializer reference<\/strong><\/a>: To make serializers work<\/li>\n<li><a href=\"https:\/\/www.django-rest-framework.org\/api-guide\/fields\/\"><strong>Serializer field reference<\/strong><\/a>: All the different serializer fields<\/li>\n<li><a href=\"https:\/\/www.django-rest-framework.org\/api-guide\/relations\/#nested-relationships\"><strong>Nested relationships<\/strong><\/a>: How to put serializers <a href=\"https:\/\/mattsegal.dev\/img\/xzibit.png\">inside of other serializers<\/a><\/li>\n<\/ul>\n<h2>Search insead of reading<\/h2>\n<p>Most documentation is not meant to be read linearly, from start to end, like a novel: most pages are too long to read. Instead, you should strategically search for what you want. Most documentation involves big lists of things, because they're so much stuff that the authors need to explain in a lot of detail. You cannot rely on brute-force reading all the content to find the info you need.<\/p>\n<p>You can use your browser's build in text search feature (<code>ctrl-f<\/code>) to quickly find the text that you need. This will save you a lot of scrolling and squinting at your screen. I use this technique all the time when browsing the Django docs. Here's a video of me finding out how to log in with Django using <code>ctrl-f<\/code>:<\/p>\n<div class=\"loom-embed\"><iframe src=\"https:\/\/www.loom.com\/embed\/cc4b030513b0406c91a1eadcd08514a2\" frameborder=\"0\" webkitallowfullscreen mozallowfullscreen allowfullscreen style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%;\"><\/iframe><\/div>\n\n<p>Here's me struggling to get past the first list by trying to read all the words with my pathetic human eyes. I genuinely did miss the \"auth\" section several times when trying to read that list manually while writing this post:<\/p>\n<div class=\"loom-embed\"><iframe src=\"https:\/\/www.loom.com\/embed\/1be42c1709334817ab3cb055ad8acf69\" frameborder=\"0\" webkitallowfullscreen mozallowfullscreen allowfullscreen style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%;\"><\/iframe><\/div>\n\n<p>Using search is how you navigate the enormous <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/contents\/\">table of contents<\/a> or the <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/topics\/auth\/default\/\">39 browser pages of authentication overview<\/a>. You're not supposed to read all that stuff, you're supposed to strategically search it. In our login example, good search terms would be \"auth\", \"login\", \"log in\" and \"user\".<\/p>\n<p>In addition, most really long pages will have a sidebar summarising all the content. If you're going to read something, read that.<\/p>\n<p><img alt=\"django sections\" src=\"https:\/\/mattsegal.dev\/img\/docs-sidebar.png\"><\/p>\n<h2>Read the source code<\/h2>\n<p>This is kind of the documentation equivalent of \"go fuck yourself\", but when you need an answer and the documentation doesn't have it, then the code is the authoratative source on how the library works. There are many library details that would be too laborious to document in full, and at some point the expectation is that if you <em>really need to know<\/em> how something works, then you should try reading the code. The <a href=\"https:\/\/github.com\/django\/django\">Django source code<\/a> is pretty well written, and the more time you spend immersed in it, the easier it will be to navigate. This isn't really advice for beginners, but if you're feeling brave, then give it a try.<\/p>\n<h2>Summary<\/h2>\n<p>The Django docs, in my opionion, really are quite good, but like most code docs, they're hard for beginners to navigate. I hope that these tips will make learning Django a more enjoyable experience for you. To summarise my tips:<\/p>\n<ul>\n<li>Identify the different sections of the documentation<\/li>\n<li>Do the tutorial first if you're not feeling confident, or at least skim read it<\/li>\n<li>Avoid the API reference early on<\/li>\n<li>Try find a how to guide for your problem<\/li>\n<li>Try find a topic overview and explanation for your topic<\/li>\n<li>Remember key references for quick lookup later<\/li>\n<li>Search the docs, don't read them like a book<\/li>\n<li>Read the source code if you're desperate<\/li>\n<\/ul>\n<p>As good as it is, the Django docs do not, and should not, tell you everything there is to know about how to use Django. At some point, you will need to turn to Django community blogs like <a href=\"https:\/\/simpleisbetterthancomplex.com\/\">Simple is Better than Complex<\/a>, YouTube videos, courses and books. When you need to deploy your Django app, you might enjoy my guide on <a href=\"https:\/\/mattsegal.dev\/simple-django-deployment.html\">Django deployment<\/a> and my overview of <a href=\"https:\/\/mattsegal.dev\/django-prod-architectures.html\">Django server setups<\/a>.<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"How to pull production data into your local Postgres database","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/restore-django-local-database.html","rel":"alternate"}},"published":"2020-06-21T12:00:00+10:00","updated":"2020-06-21T12:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-06-21:\/restore-django-local-database.html","summary":"<p>Sometimes you want to write a feature for your Django app that requires a lot of structured data that already exists in production. This happened to me recently: I needed to create a reporting tool for internal business users. The problem was that I didn't have much data in my \u2026<\/p>","content":"<p>Sometimes you want to write a feature for your Django app that requires a lot of structured data that already exists in production. This happened to me recently: I needed to create a reporting tool for internal business users. The problem was that I didn't have much data in my local database. How can I see what my reports will look like if I don't have any data?<\/p>\n<p>It's possible to generate a bunch of fake data using a management command. I've written earlier about <a href=\"https:\/\/mattsegal.dev\/django-factoryboy-dummy-data.html\">how to do this with FactoryBoy<\/a>. This approach is great for filling web pages with dummy content, but it's tedious to do if your data is highly structured and follows a bunch of implcit rules. In the case of my reporting tool, the data I wanted involved hundreds of form submissions, and each submission has dozens of answers with many different data types. Writing a script to generate data like this would haven take ages! I've also seen situations like this when working with billing systems and online stores with many product categories.<\/p>\n<p>Wouldn't it be nice if we could just get a copy of our production data and use that for local development? You could just pull the latest data from prod and work on your feature with the confidence that you have plenty of data that is structured correctly.<\/p>\n<p>In this post I'll show you a script which you can use to fetch a Postgres database backup from cloud storage and use it to populate your local Postgres database with prod data. This post builds on three previous posts of mine, which you might want to read if you can't follow the scripting in this post:<\/p>\n<ul>\n<li><a href=\"https:\/\/mattsegal.dev\/reset-django-local-database.html\">How to automatically reset your local Django database<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/postgres-backup-and-restore.html\">How to backup and restore a Postgres database<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/postgres-backup-automate.html\">How to automate your Postgres database backups<\/a><\/li>\n<\/ul>\n<p>I'm going to do all of my scripting in bash, but it's also possible to write similar scripts in PowerShell, with only a few tweaks to the syntax.<\/p>\n<h3>Starting script<\/h3>\n<p>Let's start with the \"database reset\" bash script from my <a href=\"https:\/\/mattsegal.dev\/reset-django-local-database.html\">previous post<\/a>. This script resets your local database, runs migrations and creates a local superuser for you to use. We're going to extend this script with an additional step to download and restore from our latest database backup.<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"ch\">#!\/bin\/bash<\/span>\n<span class=\"c1\"># Resets the local Django database, adding an admin login and migrations<\/span>\n<span class=\"nb\">set<\/span> -e\n<span class=\"nb\">echo<\/span> -e <span class=\"s2\">&quot;\\n&gt;&gt;&gt; Resetting the database&quot;<\/span>\n.\/manage.py reset_db --close-sessions --noinput\n\n<span class=\"c1\"># =========================================<\/span>\n<span class=\"c1\"># DOWNLOAD AND RESTORE DATABASE BACKUP HERE<\/span>\n<span class=\"c1\"># =========================================<\/span>\n\n<span class=\"nb\">echo<\/span> -e <span class=\"s2\">&quot;\\n&gt;&gt;&gt; Running migrations&quot;<\/span>\n.\/manage.py migrate\n\n<span class=\"nb\">echo<\/span> -e <span class=\"s2\">&quot;\\n&gt;&gt;&gt; Creating new superuser &#39;admin&#39;&quot;<\/span>\n.\/manage.py createsuperuser <span class=\"se\">\\<\/span>\n   --username admin <span class=\"se\">\\<\/span>\n   --email admin@example.com <span class=\"se\">\\<\/span>\n   --noinput\n\n<span class=\"nb\">echo<\/span> -e <span class=\"s2\">&quot;\\n&gt;&gt;&gt; Setting superuser &#39;admin&#39; password to 12345&quot;<\/span>\n.\/manage.py shell_plus --quiet-load -c <span class=\"s2\">&quot;<\/span>\n<span class=\"s2\">u=User.objects.get(username=&#39;admin&#39;)<\/span>\n<span class=\"s2\">u.set_password(&#39;12345&#39;)<\/span>\n<span class=\"s2\">u.save()<\/span>\n<span class=\"s2\">&quot;<\/span>\n\n<span class=\"nb\">echo<\/span> -e <span class=\"s2\">&quot;\\n&gt;&gt;&gt; Database restore finished.&quot;<\/span>\n<\/code><\/pre><\/div>\n\n<h3>Fetching the latest database backup<\/h3>\n<p>Now that we have a base script to work with, we need to fetch the latest database backup. I'm going to assume that you've followed my guide on <a href=\"https:\/\/mattsegal.dev\/postgres-backup-automate.html\">automating your Postgres database backups<\/a>.<\/p>\n<p>Let's say your database is saved in an AWS S3 bucket called <code>mydatabase-backups<\/code>, and you've saved your backups with a timestamp in the filename, like <code>postgres_mydatabase_1592731247.pgdump<\/code>. Using these two facts we can use a little bit of bash scripting to find the name of the latest backup from our S3 bucket:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># Find the latest backup file<\/span>\n<span class=\"nv\">S3_BUCKET<\/span><span class=\"o\">=<\/span>s3:\/\/mydatabase-backups\n<span class=\"nv\">LATEST_FILE<\/span><span class=\"o\">=<\/span><span class=\"k\">$(<\/span>aws s3 ls <span class=\"nv\">$S3_BUCKET<\/span> <span class=\"p\">|<\/span> awk <span class=\"s1\">&#39;{print $4}&#39;<\/span> <span class=\"p\">|<\/span> sort <span class=\"p\">|<\/span> tail -n <span class=\"m\">1<\/span><span class=\"k\">)<\/span>\n<span class=\"nb\">echo<\/span> -e <span class=\"s2\">&quot;\\nFound file <\/span><span class=\"nv\">$LATEST_FILE<\/span><span class=\"s2\"> in bucket <\/span><span class=\"nv\">$S3_BUCKET<\/span><span class=\"s2\">&quot;<\/span>\n<\/code><\/pre><\/div>\n\n<p>Once you know the name of the latest backup file, you can download it to the current directory with the <code>aws<\/code> CLI tool:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># Download the latest backup file<\/span>\naws s3 cp <span class=\"si\">${<\/span><span class=\"nv\">S3_BUCKET<\/span><span class=\"si\">}<\/span>\/<span class=\"si\">${<\/span><span class=\"nv\">LATEST_FILE<\/span><span class=\"si\">}<\/span> .\n<\/code><\/pre><\/div>\n\n<p>The <code>.<\/code> in this case refers to the current directory.<\/p>\n<h3>Restoring from the latest backup<\/h3>\n<p>Now that you've downloaded the backup file, you can apply it to your local database with <code>pg_restore<\/code>. You may need to install a Postgres client on your local machine to get access to this tool. Assuming your local Postgres credentials aren't a secret, you can just hardcode them into the script:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>pg_restore <span class=\"se\">\\<\/span>\n    --clean <span class=\"se\">\\<\/span>\n    --dbname postgres <span class=\"se\">\\<\/span>\n    --host localhost <span class=\"se\">\\<\/span>\n    --port <span class=\"m\">5432<\/span> <span class=\"se\">\\<\/span>\n    --username postgres <span class=\"se\">\\<\/span>\n    --no-owner <span class=\"se\">\\<\/span>\n    <span class=\"nv\">$LATEST_FILE<\/span>\n<\/code><\/pre><\/div>\n\n<p>In this case we use <code>--clean<\/code> to remove any existing data and we use <code>--no-owner<\/code> to ignore any commands that set ownership of objects in the database.<\/p>\n<h3>Look ma, no files!<\/h3>\n<p>You don't have to save your backup file to disk before you use it to restore your local database: you can stream the data directly from <code>aws s3 cp<\/code> to <code>pg_restore<\/code> using pipes.<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>aws s3 cp <span class=\"si\">${<\/span><span class=\"nv\">S3_BUCKET<\/span><span class=\"si\">}<\/span>\/<span class=\"si\">${<\/span><span class=\"nv\">LATEST_FILE<\/span><span class=\"si\">}<\/span> - <span class=\"p\">|<\/span> <span class=\"se\">\\<\/span>\n    pg_restore <span class=\"se\">\\<\/span>\n        --clean <span class=\"se\">\\<\/span>\n        --dbname postgres <span class=\"se\">\\<\/span>\n        --host localhost <span class=\"se\">\\<\/span>\n        --port <span class=\"m\">5432<\/span> <span class=\"se\">\\<\/span>\n        --username postgres <span class=\"se\">\\<\/span>\n        --no-owner\n<\/code><\/pre><\/div>\n\n<p>The <code>-<\/code> in this case means \"stream to stdout\", which we use so that we can pipe the data.<\/p>\n<h3>Final script<\/h3>\n<p>Here's the whole thing:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"ch\">#!\/bin\/bash<\/span>\n<span class=\"c1\"># Resets the local Django database,<\/span>\n<span class=\"c1\"># restores from latest prod backup,<\/span>\n<span class=\"c1\"># and adds an admin login and migrations<\/span>\n<span class=\"nb\">set<\/span> -e\n<span class=\"nb\">echo<\/span> -e <span class=\"s2\">&quot;\\n&gt;&gt;&gt; Resetting the database&quot;<\/span>\n.\/manage.py reset_db --close-sessions --noinput\n\n<span class=\"nb\">echo<\/span> -e <span class=\"s2\">&quot;\\nRestoring database from S3 backups&quot;<\/span>\n<span class=\"nv\">S3_BUCKET<\/span><span class=\"o\">=<\/span>s3:\/\/mydatabase-backups\n<span class=\"nv\">LATEST_FILE<\/span><span class=\"o\">=<\/span><span class=\"k\">$(<\/span>aws s3 ls <span class=\"nv\">$S3_BUCKET<\/span> <span class=\"p\">|<\/span> awk <span class=\"s1\">&#39;{print $4}&#39;<\/span> <span class=\"p\">|<\/span> sort <span class=\"p\">|<\/span> tail -n <span class=\"m\">1<\/span><span class=\"k\">)<\/span>\naws s3 cp <span class=\"si\">${<\/span><span class=\"nv\">S3_BUCKET<\/span><span class=\"si\">}<\/span>\/<span class=\"si\">${<\/span><span class=\"nv\">LATEST_FILE<\/span><span class=\"si\">}<\/span> - <span class=\"p\">|<\/span> <span class=\"se\">\\<\/span>\n    pg_restore <span class=\"se\">\\<\/span>\n        --clean <span class=\"se\">\\<\/span>\n        --dbname postgres <span class=\"se\">\\<\/span>\n        --host localhost <span class=\"se\">\\<\/span>\n        --port <span class=\"m\">5432<\/span> <span class=\"se\">\\<\/span>\n        --username postgres <span class=\"se\">\\<\/span>\n        --no-owner\n\n<span class=\"nb\">echo<\/span> -e <span class=\"s2\">&quot;\\n&gt;&gt;&gt; Running migrations&quot;<\/span>\n.\/manage.py migrate\n\n<span class=\"nb\">echo<\/span> -e <span class=\"s2\">&quot;\\n&gt;&gt;&gt; Creating new superuser &#39;admin&#39;&quot;<\/span>\n.\/manage.py createsuperuser <span class=\"se\">\\<\/span>\n   --username admin <span class=\"se\">\\<\/span>\n   --email admin@example.com <span class=\"se\">\\<\/span>\n   --noinput\n\n<span class=\"nb\">echo<\/span> -e <span class=\"s2\">&quot;\\n&gt;&gt;&gt; Setting superuser &#39;admin&#39; password to 12345&quot;<\/span>\n.\/manage.py shell_plus --quiet-load -c <span class=\"s2\">&quot;<\/span>\n<span class=\"s2\">u=User.objects.get(username=&#39;admin&#39;)<\/span>\n<span class=\"s2\">u.set_password(&#39;12345&#39;)<\/span>\n<span class=\"s2\">u.save()<\/span>\n<span class=\"s2\">&quot;<\/span>\n\n<span class=\"nb\">echo<\/span> -e <span class=\"s2\">&quot;\\n&gt;&gt;&gt; Database restore finished.&quot;<\/span>\n<\/code><\/pre><\/div>\n\n<p>You should be able to to run this over and over and over to get the latest database backup working on your local machine.<\/p>\n<h3>Other considerations<\/h3>\n<p>When talking about using production backups locally, there are two points that I think are important.<\/p>\n<p>First, production data can contain sensitive user information including names, addresses, emails and even credit card details. You need to ensure that this data is only be distributed to people who are authorised to access it, or alternatively the backups should be sanitized so the senitive data is overwritten or removed.<\/p>\n<p>Secondly, It's possible to use database backups to debug issues in production. I think it's a great method for squashing hard-to-reproduce bugs, but it shouldn't be your only way to solve production errors. Before you move onto this technique, you should first ensure you have <a href=\"https:\/\/mattsegal.dev\/file-logging-django.html\">application logging<\/a> and <a href=\"https:\/\/mattsegal.dev\/sentry-for-django-error-monitoring.html\">error monitoring<\/a> set up for your Django app, so that you don't lean on your backups as a crutch.<\/p>\n<h3>Next steps<\/h3>\n<p>If you don't already have automated prod backups, I encourage you to set that up if you have any valuable data in your Django app. Once that's done, you'll be able to use this script to pull down prod data into your local dev environment on demand.<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"How to generate lots of dummy data for your Django app","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/django-factoryboy-dummy-data.html","rel":"alternate"}},"published":"2020-06-14T12:00:00+10:00","updated":"2020-06-14T12:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-06-14:\/django-factoryboy-dummy-data.html","summary":"<p>It sucks when you're working on a Django app and all your pages are empty.\nFor example, if you're working on a forum webapp, then all your discussion boards will be empty by default:<\/p>\n<p><img alt=\"dummy-threads-empty\" src=\"https:\/\/mattsegal.dev\/dummy-threads-empty.png\"><\/p>\n<p>Manually creating enough data for your pages to look realistic is a lot of work.\nWouldn't \u2026<\/p>","content":"<p>It sucks when you're working on a Django app and all your pages are empty.\nFor example, if you're working on a forum webapp, then all your discussion boards will be empty by default:<\/p>\n<p><img alt=\"dummy-threads-empty\" src=\"https:\/\/mattsegal.dev\/dummy-threads-empty.png\"><\/p>\n<p>Manually creating enough data for your pages to look realistic is a lot of work.\nWouldn't it be nice if there was an automatic way to populate your local database with dummy data\nthat looks real? Eg. your forum app has many threads:<\/p>\n<p><img alt=\"dummy-threads\" src=\"https:\/\/mattsegal.dev\/dummy-threads-full.png\"><\/p>\n<p>Even better, wouldn't it be cool if there was an easy way to populate each thread with as many comments\nas you like?<\/p>\n<p><img alt=\"dummy-comments\" src=\"https:\/\/mattsegal.dev\/dummy-comments.png\"><\/p>\n<p>In this post I'll show you how to use <a href=\"https:\/\/factoryboy.readthedocs.io\/en\/latest\/\">Factory Boy<\/a> and a few other tricks to quickly and repeatably generate an endless amount of dummy data for your Django app. By the end of the post you'll be able to generate all your test data using a management command:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>.\/manage.py setup_test_data\n<\/code><\/pre><\/div>\n\n<p>There is example code for this blog post hosted in <a href=\"https:\/\/github.com\/MattSegal\/djdt-perf-demo\">this GitHub repo<\/a>.<\/p>\n<h3>Example application<\/h3>\n<p>In this post we'll be working with an example app that is an online forum. There are four models that we'll be working with:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># models.py<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">User<\/span><span class=\"p\">(<\/span><span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">Model<\/span><span class=\"p\">):<\/span>\n    <span class=\"sd\">&quot;&quot;&quot;A person who uses the website&quot;&quot;&quot;<\/span>\n    <span class=\"n\">name<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">CharField<\/span><span class=\"p\">(<\/span><span class=\"n\">max_length<\/span><span class=\"o\">=<\/span><span class=\"mi\">128<\/span><span class=\"p\">)<\/span>\n\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Thread<\/span><span class=\"p\">(<\/span><span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">Model<\/span><span class=\"p\">):<\/span>\n    <span class=\"sd\">&quot;&quot;&quot;A forum comment thread&quot;&quot;&quot;<\/span>\n    <span class=\"n\">title<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">CharField<\/span><span class=\"p\">(<\/span><span class=\"n\">max_length<\/span><span class=\"o\">=<\/span><span class=\"mi\">128<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">creator<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">ForeignKey<\/span><span class=\"p\">(<\/span><span class=\"n\">User<\/span><span class=\"p\">)<\/span>\n\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Comment<\/span><span class=\"p\">(<\/span><span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">Model<\/span><span class=\"p\">):<\/span>\n    <span class=\"sd\">&quot;&quot;&quot;A comment by a user on a thread&quot;&quot;&quot;<\/span>\n    <span class=\"n\">body<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">CharField<\/span><span class=\"p\">(<\/span><span class=\"n\">max_length<\/span><span class=\"o\">=<\/span><span class=\"mi\">128<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">poster<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">ForeignKey<\/span><span class=\"p\">(<\/span><span class=\"n\">User<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">thread<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">ForeignKey<\/span><span class=\"p\">(<\/span><span class=\"n\">Thread<\/span><span class=\"p\">)<\/span>\n\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Club<\/span><span class=\"p\">(<\/span><span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">Model<\/span><span class=\"p\">):<\/span>\n    <span class=\"sd\">&quot;&quot;&quot;A group of users interested in the same thing&quot;&quot;&quot;<\/span>\n    <span class=\"n\">name<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">CharField<\/span><span class=\"p\">(<\/span><span class=\"n\">max_length<\/span><span class=\"o\">=<\/span><span class=\"mi\">128<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">member<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">ManyToManyField<\/span><span class=\"p\">(<\/span><span class=\"n\">User<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n<h3>Building data with Factory Boy<\/h3>\n<p>We'll be using <a href=\"https:\/\/factoryboy.readthedocs.io\/en\/latest\/\">Factory Boy<\/a> to generate all our dummy data. It's a library that's built for automated testing, but it also works well for this use-case. Factory Boy can easily be configured to generate random but realistic data like names, emails and paragraphs by internally using the <a href=\"https:\/\/faker.readthedocs.io\/en\/master\/\">Faker<\/a> library.<\/p>\n<p>When using Factory Boy you create classes called \"factories\", which each represent a Django model. For example, for a user, you would create a factory class as follows:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># factories.py<\/span>\n<span class=\"kn\">import<\/span> <span class=\"nn\">factory<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">factory.django<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">DjangoModelFactory<\/span>\n\n<span class=\"kn\">from<\/span> <span class=\"nn\">.models<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">User<\/span>\n\n<span class=\"c1\"># Defining a factory<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">UserFactory<\/span><span class=\"p\">(<\/span><span class=\"n\">DjangoModelFactory<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">class<\/span> <span class=\"nc\">Meta<\/span><span class=\"p\">:<\/span>\n        <span class=\"n\">model<\/span> <span class=\"o\">=<\/span> <span class=\"n\">User<\/span>\n\n    <span class=\"n\">name<\/span> <span class=\"o\">=<\/span> <span class=\"n\">factory<\/span><span class=\"o\">.<\/span><span class=\"n\">Faker<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;first_name&quot;<\/span><span class=\"p\">)<\/span>\n\n<span class=\"c1\"># Using a factory with auto-generated data<\/span>\n<span class=\"n\">u<\/span> <span class=\"o\">=<\/span> <span class=\"n\">UserFactory<\/span><span class=\"p\">()<\/span>\n<span class=\"n\">u<\/span><span class=\"o\">.<\/span><span class=\"n\">name<\/span> <span class=\"c1\"># Kimberly<\/span>\n<span class=\"n\">u<\/span><span class=\"o\">.<\/span><span class=\"n\">id<\/span> <span class=\"c1\"># 51<\/span>\n\n<span class=\"c1\"># You can optionally pass in your own data<\/span>\n<span class=\"n\">u<\/span> <span class=\"o\">=<\/span> <span class=\"n\">UserFactory<\/span><span class=\"p\">(<\/span><span class=\"n\">name<\/span><span class=\"o\">=<\/span><span class=\"s2\">&quot;Alice&quot;<\/span><span class=\"p\">)<\/span>\n<span class=\"n\">u<\/span><span class=\"o\">.<\/span><span class=\"n\">name<\/span> <span class=\"c1\"># Alice<\/span>\n<span class=\"n\">u<\/span><span class=\"o\">.<\/span><span class=\"n\">id<\/span> <span class=\"c1\"># 52<\/span>\n<\/code><\/pre><\/div>\n\n<p>You can find the data types that Faker can produce by looking at the \"<a href=\"https:\/\/faker.readthedocs.io\/en\/master\/providers.html\">providers<\/a>\" that the library offers. Eg. I found \"first_name\" by reviewing the options inside the <a href=\"https:\/\/faker.readthedocs.io\/en\/master\/providers\/faker.providers.person.html\">person provider<\/a>.<\/p>\n<p>Another benefit of Factory boy is that it can be set up to generate related data using <a href=\"https:\/\/factoryboy.readthedocs.io\/en\/latest\/recipes.html#dependent-objects-foreignkey\">SubFactory<\/a>, saving you a lot of boilerplate and time. For example we can set up the <code>ThreadFactory<\/code> so that it generates a <code>User<\/code> as its creator automatically:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># factories.py<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">ThreadFactory<\/span><span class=\"p\">(<\/span><span class=\"n\">DjangoModelFactory<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">class<\/span> <span class=\"nc\">Meta<\/span><span class=\"p\">:<\/span>\n        <span class=\"n\">model<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Thread<\/span>\n\n    <span class=\"n\">creator<\/span> <span class=\"o\">=<\/span> <span class=\"n\">factory<\/span><span class=\"o\">.<\/span><span class=\"n\">SubFactory<\/span><span class=\"p\">(<\/span><span class=\"n\">UserFactory<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">title<\/span> <span class=\"o\">=<\/span> <span class=\"n\">factory<\/span><span class=\"o\">.<\/span><span class=\"n\">Faker<\/span><span class=\"p\">(<\/span>\n        <span class=\"s2\">&quot;sentence&quot;<\/span><span class=\"p\">,<\/span>\n        <span class=\"n\">nb_words<\/span><span class=\"o\">=<\/span><span class=\"mi\">5<\/span><span class=\"p\">,<\/span>\n        <span class=\"n\">variable_nb_words<\/span><span class=\"o\">=<\/span><span class=\"kc\">True<\/span>\n    <span class=\"p\">)<\/span>\n\n<span class=\"c1\"># Create a new thread<\/span>\n<span class=\"n\">t<\/span> <span class=\"o\">=<\/span> <span class=\"n\">ThreadFactory<\/span><span class=\"p\">()<\/span>\n<span class=\"n\">t<\/span><span class=\"o\">.<\/span><span class=\"n\">title<\/span>  <span class=\"c1\"># Room marriage study<\/span>\n<span class=\"n\">t<\/span><span class=\"o\">.<\/span><span class=\"n\">creator<\/span>  <span class=\"c1\"># &lt;User: Michelle&gt;<\/span>\n<span class=\"n\">t<\/span><span class=\"o\">.<\/span><span class=\"n\">creator<\/span><span class=\"o\">.<\/span><span class=\"n\">name<\/span>  <span class=\"c1\"># Michelle<\/span>\n<\/code><\/pre><\/div>\n\n<p>The ability to automatically generate related models and fake data makes Factory Boy quite powerful. It's worth taking a quick look at the <a href=\"https:\/\/factoryboy.readthedocs.io\/en\/latest\/recipes.html\">other suggested patterns<\/a> if you decide to try it out.<\/p>\n<h3>Adding a management command<\/h3>\n<p>Once you've defined all the models that you want to generate with Factory Boy, you can write a <a href=\"https:\/\/simpleisbetterthancomplex.com\/tutorial\/2018\/08\/27\/how-to-create-custom-django-management-commands.html\">management command<\/a> to automatically populate your database. This is a pretty crude script that doesn't take advantage of all of Factory Boy's features, like sub-factories, but I didn't want to spend too much time getting fancy:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># setup_test_data.py<\/span>\n<span class=\"kn\">import<\/span> <span class=\"nn\">random<\/span>\n\n<span class=\"kn\">from<\/span> <span class=\"nn\">django.db<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">transaction<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">django.core.management.base<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">BaseCommand<\/span>\n\n<span class=\"kn\">from<\/span> <span class=\"nn\">forum.models<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">User<\/span><span class=\"p\">,<\/span> <span class=\"n\">Thread<\/span><span class=\"p\">,<\/span> <span class=\"n\">Club<\/span><span class=\"p\">,<\/span> <span class=\"n\">Comment<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">forum.factories<\/span> <span class=\"kn\">import<\/span> <span class=\"p\">(<\/span>\n    <span class=\"n\">UserFactory<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">ThreadFactory<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">ClubFactory<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">CommentFactory<\/span>\n<span class=\"p\">)<\/span>\n\n<span class=\"n\">NUM_USERS<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">50<\/span>\n<span class=\"n\">NUM_CLUBS<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">10<\/span>\n<span class=\"n\">NUM_THREADS<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">12<\/span>\n<span class=\"n\">COMMENTS_PER_THREAD<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">25<\/span>\n<span class=\"n\">USERS_PER_CLUB<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">8<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Command<\/span><span class=\"p\">(<\/span><span class=\"n\">BaseCommand<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">help<\/span> <span class=\"o\">=<\/span> <span class=\"s2\">&quot;Generates test data&quot;<\/span>\n\n    <span class=\"nd\">@transaction<\/span><span class=\"o\">.<\/span><span class=\"n\">atomic<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">handle<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"o\">*<\/span><span class=\"n\">args<\/span><span class=\"p\">,<\/span> <span class=\"o\">**<\/span><span class=\"n\">kwargs<\/span><span class=\"p\">):<\/span>\n        <span class=\"bp\">self<\/span><span class=\"o\">.<\/span><span class=\"n\">stdout<\/span><span class=\"o\">.<\/span><span class=\"n\">write<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;Deleting old data...&quot;<\/span><span class=\"p\">)<\/span>\n        <span class=\"n\">models<\/span> <span class=\"o\">=<\/span> <span class=\"p\">[<\/span><span class=\"n\">User<\/span><span class=\"p\">,<\/span> <span class=\"n\">Thread<\/span><span class=\"p\">,<\/span> <span class=\"n\">Comment<\/span><span class=\"p\">,<\/span> <span class=\"n\">Club<\/span><span class=\"p\">]<\/span>\n        <span class=\"k\">for<\/span> <span class=\"n\">m<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">models<\/span><span class=\"p\">:<\/span>\n            <span class=\"n\">m<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span><span class=\"o\">.<\/span><span class=\"n\">all<\/span><span class=\"p\">()<\/span><span class=\"o\">.<\/span><span class=\"n\">delete<\/span><span class=\"p\">()<\/span>\n\n        <span class=\"bp\">self<\/span><span class=\"o\">.<\/span><span class=\"n\">stdout<\/span><span class=\"o\">.<\/span><span class=\"n\">write<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;Creating new data...&quot;<\/span><span class=\"p\">)<\/span>\n        <span class=\"c1\"># Create all the users<\/span>\n        <span class=\"n\">people<\/span> <span class=\"o\">=<\/span> <span class=\"p\">[]<\/span>\n        <span class=\"k\">for<\/span> <span class=\"n\">_<\/span> <span class=\"ow\">in<\/span> <span class=\"nb\">range<\/span><span class=\"p\">(<\/span><span class=\"n\">NUM_USERS<\/span><span class=\"p\">):<\/span>\n            <span class=\"n\">person<\/span> <span class=\"o\">=<\/span> <span class=\"n\">UserFactory<\/span><span class=\"p\">()<\/span>\n            <span class=\"n\">people<\/span><span class=\"o\">.<\/span><span class=\"n\">append<\/span><span class=\"p\">(<\/span><span class=\"n\">person<\/span><span class=\"p\">)<\/span>\n\n        <span class=\"c1\"># Add some users to clubs<\/span>\n        <span class=\"k\">for<\/span> <span class=\"n\">_<\/span> <span class=\"ow\">in<\/span> <span class=\"nb\">range<\/span><span class=\"p\">(<\/span><span class=\"n\">NUM_CLUBS<\/span><span class=\"p\">):<\/span>\n            <span class=\"n\">club<\/span> <span class=\"o\">=<\/span> <span class=\"n\">ClubFactory<\/span><span class=\"p\">()<\/span>\n            <span class=\"n\">members<\/span> <span class=\"o\">=<\/span> <span class=\"n\">random<\/span><span class=\"o\">.<\/span><span class=\"n\">choices<\/span><span class=\"p\">(<\/span>\n                <span class=\"n\">people<\/span><span class=\"p\">,<\/span>\n                <span class=\"n\">k<\/span><span class=\"o\">=<\/span><span class=\"n\">USERS_PER_CLUB<\/span>\n            <span class=\"p\">)<\/span>\n            <span class=\"n\">club<\/span><span class=\"o\">.<\/span><span class=\"n\">user<\/span><span class=\"o\">.<\/span><span class=\"n\">add<\/span><span class=\"p\">(<\/span><span class=\"o\">*<\/span><span class=\"n\">members<\/span><span class=\"p\">)<\/span>\n\n        <span class=\"c1\"># Create all the threads<\/span>\n        <span class=\"k\">for<\/span> <span class=\"n\">_<\/span> <span class=\"ow\">in<\/span> <span class=\"nb\">range<\/span><span class=\"p\">(<\/span><span class=\"n\">NUM_THREADS<\/span><span class=\"p\">):<\/span>\n            <span class=\"n\">creator<\/span> <span class=\"o\">=<\/span> <span class=\"n\">random<\/span><span class=\"o\">.<\/span><span class=\"n\">choice<\/span><span class=\"p\">(<\/span><span class=\"n\">people<\/span><span class=\"p\">)<\/span>\n            <span class=\"n\">thread<\/span> <span class=\"o\">=<\/span> <span class=\"n\">ThreadFactory<\/span><span class=\"p\">(<\/span><span class=\"n\">creator<\/span><span class=\"o\">=<\/span><span class=\"n\">creator<\/span><span class=\"p\">)<\/span>\n            <span class=\"c1\"># Create comments for each thread<\/span>\n            <span class=\"k\">for<\/span> <span class=\"n\">_<\/span> <span class=\"ow\">in<\/span> <span class=\"nb\">range<\/span><span class=\"p\">(<\/span><span class=\"n\">COMMENTS_PER_THREAD<\/span><span class=\"p\">):<\/span>\n                <span class=\"n\">commentor<\/span> <span class=\"o\">=<\/span> <span class=\"n\">random<\/span><span class=\"o\">.<\/span><span class=\"n\">choice<\/span><span class=\"p\">(<\/span><span class=\"n\">people<\/span><span class=\"p\">)<\/span>\n                <span class=\"n\">CommentFactory<\/span><span class=\"p\">(<\/span>\n                    <span class=\"n\">user<\/span><span class=\"o\">=<\/span><span class=\"n\">commentor<\/span><span class=\"p\">,<\/span>\n                    <span class=\"n\">thread<\/span><span class=\"o\">=<\/span><span class=\"n\">thread<\/span>\n                <span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n<p>Using the <code>transaction.atomic<\/code> decorator makes a big difference in the runtime of this script, since it bundles up 100s of queries and submits them in one go.<\/p>\n<h3>Images<\/h3>\n<p>If you need dummy images for your website as well then there are a lot of great free tools online to help. I use <a href=\"https:\/\/api.adorable.io\">adorable.io<\/a> for dummy profile pics and <a href=\"https:\/\/picsum.photos\/\">Picsum<\/a> or <a href=\"https:\/\/unsplash.com\/developers\">Unsplash<\/a> for larger pictures like this one: <a href=\"https:\/\/picsum.photos\/700\/500\">https:\/\/picsum.photos\/700\/500<\/a>.<\/p>\n<p><img alt=\"picsum-example\" src=\"https:\/\/picsum.photos\/700\/500\"><\/p>\n<h3>Next steps<\/h3>\n<p>Hopefully this post helps you spin up a lot of fake data for your Django app very quickly.\nIf you enjoy using Factory Boy to generate your dummy data, then you also might like incorporating it into your unit tests.<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"How to automatically reset your local Django database","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/reset-django-local-database.html","rel":"alternate"}},"published":"2020-06-13T12:00:00+10:00","updated":"2020-06-13T12:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-06-13:\/reset-django-local-database.html","summary":"<p>Sometimes when you're working on a Django app you want a fresh start. You want to nuke all of the data in your local database and start again from scratch. Maybe you ran some migrations that you don't want to keep, or perhaps there's some test data that you want \u2026<\/p>","content":"<p>Sometimes when you're working on a Django app you want a fresh start. You want to nuke all of the data in your local database and start again from scratch. Maybe you ran some migrations that you don't want to keep, or perhaps there's some test data that you want to get rid of. This kind of problem doesn't crop up very often, but when it does it's <em>super<\/em> annoying to do it manually over and over.<\/p>\n<p>In this post I'll show you small script that you can use to reset your local Django database. It completely automates deleting the old data, running migrations and setting up new users. I've written the script in <code>bash<\/code> but most of it will also work in <code>powershell<\/code> or <code>cmd<\/code> with only minor changes.<\/p>\n<p>For those of you who hate reading, the full script is near the bottom.<\/p>\n<h3>Resetting the database<\/h3>\n<p>We're going to reset our local database with the <a href=\"https:\/\/django-extensions.readthedocs.io\/en\/latest\/installation_instructions.html\">django-extensions<\/a> package, which provides a nifty little helper command called <code>reset_db<\/code>. This command destroys and recreates your Django app's database.<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>.\/manage.py reset_db\n<\/code><\/pre><\/div>\n\n<p>I like to add the <code>--noinput<\/code> flag so the script does not ask me for confirmation, and the <code>--close-sessions<\/code> flag if I'm using PostgreSQL locally so that the command does not fail if my Django app is connected the database at the same time.<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>.\/manage.py reset_db --noinput --close-sessions\n<\/code><\/pre><\/div>\n\n<p>This is is a good start, but now we have no migrations, users or any other data in our database. We need to add some data back in there before we can start using the app again.<\/p>\n<h3>Running migrations<\/h3>\n<p>Before you do anything else it's important to run migrations so that all your database tables are set up correctly:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>.\/manage.py migrate\n<\/code><\/pre><\/div>\n\n<h3>Creating an admin user<\/h3>\n<p>You want to have a superuser set up so you can log into the Django admin. It's nice when a script guarantees that your superuser always has the same username and password. The first part of creating a superuser is pretty standard:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>.\/manage.py createsuperuser <span class=\"se\">\\<\/span>\n   --username admin <span class=\"se\">\\<\/span>\n   --email admin@example.com <span class=\"se\">\\<\/span>\n   --noinput\n<\/code><\/pre><\/div>\n\n<p>Now we want to set the admin user's password to something easy to remember, like \"12345\". This isn't a security risk because it's just for local development. This step involves a little more scripting trickery. Here we can use <code>shell_plus<\/code>, which is an enhanced Django shell provided by django-extensions. The <code>shell_plus<\/code> command will automatically import all of our models, which means we can write short one liners like this one, which prints the number of Users in the database:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>.\/manage.py shell_plus --quiet-load -c <span class=\"s2\">&quot;print(User.objects.count())&quot;<\/span>\n<span class=\"c1\"># 13<\/span>\n<\/code><\/pre><\/div>\n\n<p>Using this method we can grab our admin user and set their password:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>.\/manage.py shell_plus --quiet-load -c <span class=\"s2\">&quot;<\/span>\n<span class=\"s2\">u = User.objects.get(username=&#39;admin&#39;)<\/span>\n<span class=\"s2\">u.set_password(&#39;12345&#39;)<\/span>\n<span class=\"s2\">u.save()<\/span>\n<span class=\"s2\">&quot;<\/span>\n<\/code><\/pre><\/div>\n\n<h3>Setting up new data<\/h3>\n<p>There might be a little bit of data that you want to set up every time you reset your database. For example, in one app I run, I want to ensure that there is always a <code>SlackMessage<\/code> model that has a <code>SlackChannel<\/code>. We can set up this data in the same way we set up the admin user's password:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>.\/manage.py shell_plus --quiet-load -c <span class=\"s2\">&quot;<\/span>\n<span class=\"s2\">c = SlackChannel.objects.create(name=&#39;Test Alerts&#39;)<\/span>\n<span class=\"s2\">SlackMessage.objects.create(channel=c)<\/span>\n<span class=\"s2\">&quot;<\/span>\n<\/code><\/pre><\/div>\n\n<p>If you need to set up a <em>lot<\/em> of data then there are options like <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/howto\/initial-data\/\">fixtures<\/a> or tools like <a href=\"https:\/\/factoryboy.readthedocs.io\/en\/latest\/\">Factory Boy<\/a> (which I heartily recommend). If you only need to do a few lines of scripting to create your data, then you can include them in this script. If your development data setup is very complicated, then I recommend putting all the setup code into a custom management command.<\/p>\n<h3>The final script<\/h3>\n<p>This is the script that you can use to reset your local Django database:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"ch\">#!\/bin\/bash<\/span>\n<span class=\"c1\"># Resets the local Django database, adding an admin login and migrations<\/span>\n<span class=\"nb\">set<\/span> -e\n<span class=\"nb\">echo<\/span> -e <span class=\"s2\">&quot;\\n&gt;&gt;&gt; Resetting the database&quot;<\/span>\n.\/manage.py reset_db --close-sessions --noinput\n\n<span class=\"nb\">echo<\/span> -e <span class=\"s2\">&quot;\\n&gt;&gt;&gt; Running migrations&quot;<\/span>\n.\/manage.py migrate\n\n<span class=\"nb\">echo<\/span> -e <span class=\"s2\">&quot;\\n&gt;&gt;&gt; Creating new superuser &#39;admin&#39;&quot;<\/span>\n.\/manage.py createsuperuser <span class=\"se\">\\<\/span>\n   --username admin <span class=\"se\">\\<\/span>\n   --email admin@example.com <span class=\"se\">\\<\/span>\n   --noinput\n\n<span class=\"nb\">echo<\/span> -e <span class=\"s2\">&quot;\\n&gt;&gt;&gt; Setting superuser &#39;admin&#39; password to 12345&quot;<\/span>\n.\/manage.py shell_plus --quiet-load -c <span class=\"s2\">&quot;<\/span>\n<span class=\"s2\">u=User.objects.get(username=&#39;admin&#39;)<\/span>\n<span class=\"s2\">u.set_password(&#39;12345&#39;)<\/span>\n<span class=\"s2\">u.save()<\/span>\n<span class=\"s2\">&quot;<\/span>\n\n<span class=\"c1\"># Any extra data setup goes here.<\/span>\n\n<span class=\"nb\">echo<\/span> -e <span class=\"s2\">&quot;\\n&gt;&gt;&gt; Database restore finished.&quot;<\/span>\n<\/code><\/pre><\/div>\n\n<h3>Other methods<\/h3>\n<p>It's good to note that what I'm proposing is the \"nuclear option\": purge everything and restart from scratch. There are also some more precise methods available for managing your local database:<\/p>\n<ul>\n<li>If you just want to reverse some particular migrations, then you can use the <code>migrate<\/code> command <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/topics\/migrations\/#reversing-migrations\">as documented here<\/a>.<\/li>\n<li>If you just want to delete all your data and you don't care about re-applying the migrations, then the <code>flush<\/code> management command, <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/ref\/django-admin\/#flush\">documented here<\/a> will take care of that.<\/li>\n<\/ul>\n<h3>Docker environments<\/h3>\n<p>If you're running your local Django app in a Docker container via <code>docker-compose<\/code>, then this process is a little bit more tricky, but it's not too much more complicated. You just need to add two commands to your script.<\/p>\n<p>First you want a command to kill all running containers, which I do because I'm superstitious and don't trust that <code>reset_db<\/code> will actually close all database connections:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"k\">function<\/span> stop_docker <span class=\"o\">{<\/span>\n    <span class=\"nb\">echo<\/span> -e <span class=\"s2\">&quot;\\nStopping all running Docker containers&quot;<\/span>\n    <span class=\"c1\"># Ensure that no containers automatically restart<\/span>\n    docker update --restart<span class=\"o\">=<\/span>no <span class=\"sb\">`<\/span>docker ps -q<span class=\"sb\">`<\/span>\n    <span class=\"c1\"># Kill everything<\/span>\n    docker <span class=\"nb\">kill<\/span> <span class=\"sb\">`<\/span>docker ps -q<span class=\"sb\">`<\/span>\n<span class=\"o\">}<\/span>\n<\/code><\/pre><\/div>\n\n<p>We also want a shorthand way to run commands inside your docker environment. Let's say you are working with a compose file located at <code>docker\/docker-compose.local.yml<\/code> and your Django app's container is called <code>web<\/code>. Then you can run your commands inside the container as follows:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"k\">function<\/span> run_docker <span class=\"o\">{<\/span>\n    docker-compose -f docker\/docker-compose.local.yml run --rm web <span class=\"nv\">$@<\/span>\n<span class=\"o\">}<\/span>\n<\/code><\/pre><\/div>\n\n<p>Now we can just prefix <code>run_docker<\/code> to all the management commands we run. For example:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># Without Docker<\/span>\n.\/manage.py reset_db --close-sessions --noinput\n<span class=\"c1\"># With Docker<\/span>\nrun_docker .\/manage.py reset_db --close-sessions --noinput\n<\/code><\/pre><\/div>\n\n<p>I will note that this <code>run_docker<\/code> shortcut can act a little weird when you're passing strings to <code>shell_plus<\/code>. You might need to experiment with different methods of escaping whitespace etc.<\/p>\n<h3>Conclusion<\/h3>\n<p>Hopefully this script will save you some time when you're working on your Django app. If you're interested in more Django-related database stuff then you might enjoy reading about how to <a href=\"https:\/\/mattsegal.dev\/postgres-backup-and-restore.html\">back up and restore a Postgres database<\/a> and then how to <a href=\"https:\/\/mattsegal.dev\/postgres-backup-automate.html\">fully automate your prod backup process<\/a>.<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"A tour of Django server setups","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/django-prod-architectures.html","rel":"alternate"}},"published":"2020-05-25T12:00:00+10:00","updated":"2020-05-25T12:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-05-25:\/django-prod-architectures.html","summary":"<p>If you haven't deployed a lot of Django apps, then you might wonder:\nhow do professionals put Django apps on the internet? What does Django typically look like when it's running in production?\nYou might even be thinking <em>what the hell is <a href=\"https:\/\/www.techopedia.com\/definition\/8989\/production-environment\">production<\/a>?<\/em><\/p>\n<p>Before I started working a developer there \u2026<\/p>","content":"<p>If you haven't deployed a lot of Django apps, then you might wonder:\nhow do professionals put Django apps on the internet? What does Django typically look like when it's running in production?\nYou might even be thinking <em>what the hell is <a href=\"https:\/\/www.techopedia.com\/definition\/8989\/production-environment\">production<\/a>?<\/em><\/p>\n<p>Before I started working a developer there was just a fuzzy cloud in my head where the knowledge of production infrastructure should be.\nIf there's a fuzzy cloud in your head, let's fix it.\nThere are many ways to extend a Django server setup to achieve better performance, cost-effectiveness and reliability.\nThis post will take you on a tour of some common Django server setups, from the most simple and basic to the more complex and powerful.\nI hope it will build up your mental model of how Django is hosted in production, piece-by-piece.<\/p>\n<h2>Your local machine<\/h2>\n<p>Let's start by reviewing a Django setup that you are alreay familiar with: your local machine.\nGoing over this will be a warm-up for later sections.\nWhen you run Django locally, you have:<\/p>\n<ul>\n<li>Your web browser (Chrome, Safari, Firefox, etc)<\/li>\n<li>Django running with the runserver management command<\/li>\n<li>A SQLite database sitting in your project folder<\/li>\n<\/ul>\n<p><img alt=\"local server setup\" src=\"https:\/\/mattsegal.dev\/django-prod-architecture\/local-server.png\"><\/p>\n<p>Pretty simple right? Next let's look at something similar, but deployed to a web server.<\/p>\n<h2>Simplest possible webserver<\/h2>\n<p>The simplest Django web server you can setup is very similar to your local dev environment.\nMost professional Django devs don't use a basic setup like this for their production environments. It works perfectly fine, but it has some limitations that we'll discuss later.\nIt looks like this:<\/p>\n<p><img alt=\"simple server setup\" src=\"https:\/\/mattsegal.dev\/django-prod-architecture\/simple-server.png\"><\/p>\n<p>Typically people run Django on a Linux virtual machine, often using the Ubuntu distribution.\nThe virtual machine is hosted by a cloud provider like <a href=\"https:\/\/aws.amazon.com\/\">Amazon<\/a>, <a href=\"https:\/\/cloud.google.com\/gcp\/\">Google<\/a>, <a href=\"https:\/\/azure.microsoft.com\/en-au\/\">Azure<\/a>, <a href=\"https:\/\/www.digitalocean.com\/\">DigitalOcean<\/a> or <a href=\"https:\/\/www.linode.com\/\">Linode<\/a>.<\/p>\n<p>Instead of using runserver, you should use a WSGI server like <a href=\"https:\/\/gunicorn.org\/\">Gunicorn<\/a> to run your Django app.\nI go into more detail on why you shouldn't use runserver in production, and explain WSGI <a href=\"https:\/\/mattsegal.dev\/simple-django-deployment-2.html#wsgi\">here<\/a>.\nOtherwise, not that much is different from your local machine: you can still use SQLite as the database (<a href=\"https:\/\/mattsegal.dev\/simple-django-deployment-2.html#sqlite\">more here<\/a>).<\/p>\n<p>This is the bare bones of the setup. There are a few other details that you'll need to manage like <a href=\"https:\/\/mattsegal.dev\/dns-for-noobs.html\">setting up DNS<\/a>, virtual environments, babysitting Gunicorn with a process supervisor like <a href=\"https:\/\/mattsegal.dev\/simple-django-deployment-4.html\">Supervisord<\/a> or how to serve static files with <a href=\"http:\/\/whitenoise.evans.io\/en\/stable\/\">Whitenoise<\/a>. If you're interested in a more complete guide on how to set up a simple server like this, I wrote <a href=\"https:\/\/mattsegal.dev\/simple-django-deployment.html\">a guide<\/a> that explains how to deploy Django.<\/p>\n<h2>Typical standalone webserver<\/h2>\n<p>Let's go over an environment that a professional Django dev might set up in production when using a single server.\nIt's not the exact setup that everyone will always use, but the structure is very common.<\/p>\n<p><img alt=\"typical server setup\" src=\"https:\/\/mattsegal.dev\/django-prod-architecture\/typical-server.png\"><\/p>\n<p>Some things are the same as the simple setup above: it's still a Linux virtual machine with Django being run by Gunicorn.\nThere are three main differences:<\/p>\n<ul>\n<li>SQLite has been replaced by a different database, <a href=\"https:\/\/www.postgresql.org\/\">PostgreSQL<\/a><\/li>\n<li>A <a href=\"https:\/\/www.nginx.com\/\">NGINX<\/a> web server is now sitting in-front of Gunicorn in a <a href=\"https:\/\/www.nginx.com\/resources\/glossary\/reverse-proxy-server\/\">reverse-proxy<\/a> setup<\/li>\n<li>Static files are now being served from outside of Django<\/li>\n<\/ul>\n<p>Why did we swap SQLite for PostgreSQL? In general Postgres is a litte more advanced and full featured. For example, Postgres can handle multiple writes at the same\ntime, while SQLite can't.<\/p>\n<p>Why did we add NGINX to our setup? NGINX is a dedicated webserver which provides extra features and performance improvements\nover just using Gunicorn to serve web requests. For example we can use NGINX to directly serve our app's static and media files more efficiently. NGINX can also be configured to a lot of other useful things, like encrypt your web traffic using HTTPS and compress your files to make your site faster. NGINX is the web server that is most commonly combined with Django, but there are also alternatives like the <a href=\"https:\/\/httpd.apache.org\/\">Apache HTTP server<\/a> and <a href=\"https:\/\/docs.traefik.io\/\">Traefik<\/a>.<\/p>\n<p>It's important to note that everything here lives on a single server, which means that if the server goes away, so does all your data, <a href=\"https:\/\/mattsegal.dev\/postgres-backup-and-restore.html\">unless you have backups<\/a>.\nThis data includes your Django tables, which are stored in Postgres, and files uploaded by users, which will be stored in the <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/ref\/settings\/#media-root\">MEDIA_ROOT<\/a> folder, somewhere on your filesystem. Having only one server also means that if your server restarts or shuts off, so does your website. This is OK for smaller projects, but it's not acceptable for big sites like StackOverflow or Instagram, where the cost of downtime is very high.<\/p>\n<div class=\"ui divider\" style=\"margin: 1.5em 0;\"><\/div>\n<form action=\"https:\/\/dev.us19.list-manage.com\/subscribe\/post?u=e7a1ec466f7bb1732dbd23fc7&amp;id=ec345473bd\" method=\"post\" name=\"mc-embedded-subscribe-form\" target=\"_blank\" style=\"text-align: center; padding-bottom: 1em;\" novalidate>\n  <h3 class=\"subscribe-cta\">Get alerted when I publish new blog posts<\/h3>\n  <div class=\"ui fluid action input subscribe\">\n    <input\n      type=\"email\"\n      value=\"\"\n      name=\"EMAIL\"\n      placeholder=\"Enter your email address\"\n    \/>\n    <button class=\"ui primary button\" type=\"submit\" name=\"subscribe\">\n      Subscribe\n    <\/button>\n  <\/div>\n  <div style=\"position: absolute; left: -5000px;\" aria-hidden=\"true\">\n    <input\n      type=\"text\"\n      name=\"b_e7a1ec466f7bb1732dbd23fc7_ec345473bd\"\n      tabindex=\"-1\"\n      value=\"\"\n    \/>\n  <\/div>\n<\/form>\n<div class=\"ui divider\" style=\"margin: 1.5em 0;\"><\/div>\n\n<h2>Single webserver with multiple apps<\/h2>\n<p>Once you start using NGINX and PostgreSQL, you can run multiple Django apps on the same machine.\nYou can save money on hosting fees by packing multiple apps onto a single server rather than paying for a separate server for each app. This setup also allows you to re-use some of the services\nand configurations that you've already set up.<\/p>\n<p>NGINX is able to route incoming HTTP requests to different apps based on the domain name, and Postgres can host multiple databases on a single machine.\nFor example, I use a single server to host some of my personal Django projects: <a href=\"http:\/\/mattslinks.xyz\/\">Matt's Links<\/a>, <a href=\"http:\/\/memories.ninja\/\">Memories Ninja<\/a> and <a href=\"https:\/\/www.blogreader.com.au\/\">Blog Reader<\/a><\/p>\n<p><img alt=\"multi-app server setup\" src=\"https:\/\/mattsegal.dev\/django-prod-architecture\/multi-app-server.png\"><\/p>\n<p>I've omitted the static files for simplicity. Note that having multiple apps on one server saves you hosting costs, but there are downsides: restarting the server restarts all of your apps.<\/p>\n<h3 id=\"worker\">Single webserver with a worker<\/h3>\n\n<p>Some web apps need to do things other than just <a href=\"https:\/\/www.codecademy.com\/articles\/what-is-crud\">CRUD<\/a>. For example, my website <a href=\"https:\/\/www.blogreader.com.au\/\">Blog Reader<\/a> needs to scrape <a href=\"https:\/\/slatestarcodex.com\/2020\/04\/24\/employer-provided-health-insurance-delenda-est\/\">text<\/a> from a website and then send it to an Amazon API to be translated into <a href=\"https:\/\/media.blogreader.com.au\/media\/043dcf9fe4c1df539468000cb97af1d7.mp3\">audio files<\/a>. Another common example is \"thumbnailing\", where you upload a huge 5MB image file to Facebook and they downsize it into a crappy 120kB JPEG. These kinds of tasks do not happen inside a Django view, because they take too long to run. Instead they have to happen \"offline\", in a separate worker process, using tools like <a href=\"http:\/\/www.celeryproject.org\/\">Celery<\/a>, <a href=\"https:\/\/huey.readthedocs.io\/en\/latest\/django.html\">Huey<\/a>, <a href=\"https:\/\/github.com\/rq\/django-rq\">Django-RQ<\/a> or <a href=\"https:\/\/django-q.readthedocs.io\/en\/latest\/\">Django-Q<\/a>. All these tools provide you with a way to run tasks outside of Django views and do more complicated things, like co-ordinate multiple tasks and run them on schedules.<\/p>\n<p>All of these tools follow a similar pattern: tasks are dispatched by Django and put in a queue where they wait to be executed. This queue is managed by a service called a \"broker\", which keeps track of all the tasks that need to be done. Common brokers for Django tasks are Redis and RabbitMQ. A worker process, which uses the same codebase as your Django app, pulls tasks out the broker and runs them.<\/p>\n<p><img alt=\"worker server setup\" src=\"https:\/\/mattsegal.dev\/django-prod-architecture\/worker-server.png\"><\/p>\n<p>If you haven't worked with task queues before then it's not immediately obvious how this all works, so let me give an example. You want to upload a 2MB <a href=\"https:\/\/memories-ninja-prod.s3-ap-southeast-2.amazonaws.com\/original\/7e26334177b6ee7d5ab4c21f7149190e.jpeg\">photo of your breakfast<\/a> from your phone to a Django site. To optimise image loading performance, the Django site will turn that 2MB photo upload into a 70kB <a href=\"https:\/\/memories-ninja-prod.s3.amazonaws.com\/display\/7e26334177b6ee7d5ab4c21f7149190e.jpeg\">display image<\/a> and a smaller <a href=\"https:\/\/memories-ninja-prod.s3.amazonaws.com\/thumbnail\/7e26334177b6ee7d5ab4c21f7149190e.jpeg\">thumbnail image<\/a>. So this is what happenes:<\/p>\n<ul>\n<li>A user uploads a photo to a Django view, which saves the original photo to the filesystem and updates the database to show that the file has been received<\/li>\n<li>The view also pushes a thumbnailing task to the task broker<\/li>\n<li>The broker receives the task and puts it in a queue, where it waits to be executed<\/li>\n<li>The worker asks the broker for the next task and the broker sends the thumbnailing tasks<\/li>\n<li>The worker reads the task description and runs some Python function, which reads the original image from the filesystem, creates the smaller thumbnail images, saves them and then updates the database to show that the thumbnailing is complete<\/li>\n<\/ul>\n<p>If you want to learn more about this stuff, I've written guides for getting started with <a href=\"https:\/\/mattsegal.dev\/offline-tasks.html\">offline tasks<\/a> and <a href=\"https:\/\/mattsegal.dev\/simple-scheduled-tasks.html\">scheduled tasks<\/a> with Django Q.<\/p>\n<h2>Single webserver with a cache<\/h2>\n<p>Sometimes you'll want to <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/topics\/cache\/\">use a cache<\/a> to store data for a short time. For example, caches are commonly used when you have some data that was expensive to pull from the database or an API and you want to re-use it for a little while. <a href=\"https:\/\/redis.io\/\">Redis<\/a> and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Memcached\">Memcached<\/a> are both popular cache services that are used in production with Django. It's not a very complicated setup.<\/p>\n<p><img alt=\"cache on server setup\" src=\"https:\/\/mattsegal.dev\/django-prod-architecture\/cache-on-server.png\"><\/p>\n<h2>Single webserver with Docker<\/h2>\n<p>If you've heard of <a href=\"https:\/\/www.docker.com\/\">Docker<\/a> before you might be wondering where it factors into these setups.\nIt's a great tool for creating consistent programming environments, but it doesn't actually change how any of this works too much.\nMost of the setups I've described would work basically the same way... except everything is inside a Docker container.<\/p>\n<p>For example, if you were running multiple Django apps on one server and you wanted to use Docker containers, then\nyou might do something like this using <a href=\"https:\/\/docs.docker.com\/engine\/swarm\/\">Docker Swarm<\/a>:<\/p>\n<p><img alt=\"docker on server setup\" src=\"https:\/\/mattsegal.dev\/django-prod-architecture\/swarm-server.png\"><\/p>\n<p>As you can see it's not such a different structure compared to what we were doing before Docker.\nThe containers are just wrappers around the services that we were already running.\nPutting things inside of Docker containers doesn't really change how all the services talk to each other.\nIf you really wanted to you could wrap Docker containers around more things like NGINX, the database, a Redis cache, whatever.\nThis is why I think it's valuable to learn how to deploy Django without Docker first.\nThat said, you can do some more complicated setups with Docker containers, which we'll get into later.<\/p>\n<h2>External services<\/h2>\n<p>So far I've been showing you server setups with just one virtual machine running Ubuntu.\nThis is the simplest setup that you can use, but it has limitations: there are some things that\nyou might need that a single server can't give you. In this section I'm going to walk you through\nhow we can break apart our single server into more advanced setups.<\/p>\n<p>If you've studied programming you might have read about <a href=\"https:\/\/en.wikipedia.org\/wiki\/Separation_of_concerns\">separation of concerns<\/a>, the\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/Single-responsibility_principle\">single responsibility principle<\/a> and\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/Model%E2%80%93view%E2%80%93controller\">model-view-controller (MVC)<\/a>.\nA lot of the changes that we're going to make will have a similar kind of vibe: we're going to split up our services\ninto smaller, more specialised units, based on their \"responsibilities\".\nWe're going to pull apart our services bit-by-bit until there's nothing left.\nJust a note: you might not need to do this for your services, this is just an overview of what you <em>could<\/em> do.<\/p>\n<h2>External services - database<\/h2>\n<p>The first thing you'd want to pull off of our server is the database. This involves putting PostgreSQL onto its own virtual machine.\nYou can set this up yourself or pay a little extra for an off-the-shelf service like <a href=\"https:\/\/aws.amazon.com\/rds\/\">Amazon RDS<\/a>.<\/p>\n<p><img alt=\"postgres on server setup\" src=\"https:\/\/mattsegal.dev\/django-prod-architecture\/postgres-external.png\"><\/p>\n<p>There are a couple of reasons that you'd want to put the database on its own server:<\/p>\n<ul>\n<li>You might have multiple apps on different servers that depend on the same database<\/li>\n<li>Your database performance will not be impacted by \"noisy neighbours\" eating up CPU, RAM or disk space on the same machine<\/li>\n<li>You've moved your precious database away from your Django web server, which means you can delete and re-create your Django app's server with less concern<\/li>\n<li><em>mumble muble security mumble<\/em><\/li>\n<\/ul>\n<p>Using an off-the-shelf option like AWS RDS is attractive because it reduces the amount of admin work that you need to run your database server.\nIf you're a backend web developer with a lot of work to do and more money than time then this is a good move.<\/p>\n<h2>External services - object storage<\/h2>\n<p>It is common to push file storage off the web server into \"object storage\", which is basically a filesystem behind a nice API. This is often done using <a href=\"https:\/\/django-storages.readthedocs.io\/en\/latest\/\">django-storages<\/a>, which I enjoy using. Object storage is usually used for user-uploaded \"media\" such as documents, photos and videos. I use AWS S3 (Simple Storage Service) for this, but every big cloud hosting provider has some sort of \"object storage\" offering.<\/p>\n<p><img alt=\"AWS S3 setup\" src=\"https:\/\/mattsegal.dev\/django-prod-architecture\/files-external-revised.png\"><\/p>\n<p>There are a few reasons why this is a good idea<\/p>\n<ul>\n<li>You've moved all of your app's state (files, database) off of your server, so now you can move, destroy and re-create the Django server with no data loss<\/li>\n<li>File downloads hit the object storage service, rather than your server, meaning you can scale your file downloads more easily<\/li>\n<li>You don't need to worry about any filesystem admin, like running out of disk space<\/li>\n<li>Multiple servers can easily share the same set of files<\/li>\n<\/ul>\n<p>Hopefully you see a theme here, we're taking shit we don't care about and making it someone else's problem.\nPaying someone else to do the work of managing our files and database leaves us more free time to work on more important things.<\/p>\n<h2>External services - web server<\/h2>\n<p>You can also run your \"web server\" (NGINX) on a different virtual machine to your \"app server\" (Gunicorn + Django):<\/p>\n<p><img alt=\"nginx external setup\" src=\"https:\/\/mattsegal.dev\/django-prod-architecture\/nginx-1-external.png\"><\/p>\n<p>This seems kind of pointless though, why would you bother? Well, for one, you might have multiple identical app servers set up for redundancy and to handle high traffic, and NGINX can act as a <a href=\"https:\/\/www.nginx.com\/resources\/glossary\/load-balancing\/\">load balancer<\/a> between the different servers.<\/p>\n<p><img alt=\"nginx external setup 2\" src=\"https:\/\/mattsegal.dev\/django-prod-architecture\/nginx-2-external.png\"><\/p>\n<p>You could also replace NGINX with an off-the-shelf load balancer like an AWS Elastic Load Balancer or something similar.<\/p>\n<p>Note how putting our services on their own servers allows us to scale them out over multiple virtual machines. We couldn't run our Django app on three servers at the same time if we also had three copies of our filesystem and three databases.<\/p>\n<h2>External services - task queue<\/h2>\n<p>You can also push your \"offline task\" services onto their own servers. Typically the broker service would get its own machine and the worker would live on another:<\/p>\n<p><img alt=\"worker external setup\" src=\"https:\/\/mattsegal.dev\/django-prod-architecture\/worker-1-external.png\"><\/p>\n<p>Splitting your worker onto its own server is useful because:<\/p>\n<ul>\n<li>You can protect your Django web app from \"noisy neighbours\": workers which are hogging all the RAM and CPU<\/li>\n<li>You can give the worker server extra resources that it needs: CPU, RAM, or access to a GPU<\/li>\n<li>You can now make changes to the worker server without risking damage to the task queue or the web server<\/li>\n<\/ul>\n<p>Now that you've split things up, you can also scale out your workers to run more tasks in parallel:<\/p>\n<p><img alt=\"worker external setup 2\" src=\"https:\/\/mattsegal.dev\/django-prod-architecture\/worker-2-external.png\"><\/p>\n<p>You could potentially swap our your self-managed broker (Redis or RabbitMQ) for a managed queue like <a href=\"https:\/\/aws.amazon.com\/sqs\/\">Amazon SQS<\/a>.<\/p>\n<h2>External services - final form<\/h2>\n<p>If you went totally all-out, your Django app could be set up like this:<\/p>\n<p><img alt=\"fully external setup\" src=\"https:\/\/mattsegal.dev\/django-prod-architecture\/full-external.png\"><\/p>\n<p>As you can see, you can go pretty crazy splitting up all the parts of your Django app and spreading across multiple servers.\nThere are many upsides to this, but the downside is that you now have mutiple servers to provision, update, monitor and maintain.\nSometimes the extra complexity is well worth or and sometimes it's a waste of your time. That said, there are many benefits to this setup:<\/p>\n<ul>\n<li>Your web and worker servers are completely replaceable, you can destroy, create and update them without affecting uptime at all<\/li>\n<li>You can now do <a href=\"https:\/\/martinfowler.com\/bliki\/BlueGreenDeployment.html\">blue-green deployments<\/a> with zero web app downtime<\/li>\n<li>Your files and database are easily shared between multiple servers and applications<\/li>\n<li>You can provision different sized servers for their different workloads<\/li>\n<li>You can swap out your self-managed servers for managed infrastructure, like moving your task broker to AWS SQS, or your database to AWS RDS<\/li>\n<li>You can now autoscale your servers (more on this later)<\/li>\n<\/ul>\n<p>When you have complicated infrastructure like this you need to start automating your infrastructure setup and server config.\nIt's just not feasible to manage this stuff manually once your setup has this many moving parts. I recorded a talk\non <a href=\"https:\/\/mattsegal.dev\/intro-config-management.html\">configuration management<\/a> that introduces these concepts.\nYou'll need to start looking into tools like <a href=\"https:\/\/www.ansible.com\/\">Ansible<\/a> and <a href=\"https:\/\/www.packer.io\/\">Packer<\/a> to configure your virtual machines,\nand tools like <a href=\"https:\/\/www.terraform.io\/\">Terraform<\/a> or <a href=\"https:\/\/aws.amazon.com\/cloudformation\/\">CloudFormation<\/a> to configure your cloud services.<\/p>\n<h2>Auto scaling groups<\/h2>\n<p>You've already seen how you can have multiple web servers running the same app, or multiple worker servers all pulling tasks from a queue.\nThese servers cost money, dollars per hour, and it can get very expensive to run more servers than you need.<\/p>\n<p>This is where <a href=\"https:\/\/aws.amazon.com\/autoscaling\/\">autoscaling<\/a> comes in. You can setup your cloud services to use some sort of trigger, such as virtual machine CPU usage,\nto automatically create new virtual machines from an image and add them to an autoscaling group.<\/p>\n<p>Let's use our task worker servers as an example. If you have a thumbnailing service that turns <a href=\"https:\/\/memories-ninja-prod.s3-ap-southeast-2.amazonaws.com\/original\/7e26334177b6ee7d5ab4c21f7149190e.jpeg\">big uploaded photos<\/a> into <a href=\"https:\/\/memories-ninja-prod.s3.amazonaws.com\/thumbnail\/7e26334177b6ee7d5ab4c21f7149190e.jpeg\">smaller photos<\/a> then one server should be able to handle\ndozens of file uploads per second. What if during some periods of the day, like around 6pm after work, you saw file uploads spike from dozens per second to <em>thousands<\/em> per second? Then you'd need more servers!\nWith an autoscaling setup, the CPU usage on your worker servers would spike, triggering the creation of more and more worker servers, until you had enough to handle all the uploads.\nWhen the rate of file uploads drops, the extra servers would be automatically destroyed, so you aren't always paying for them.<\/p>\n<h2>Container clusterfuck<\/h2>\n<p>There is a whole world of container fuckery that I haven't covered in much detail, because:<\/p>\n<ul>\n<li>I don't know it very well<\/li>\n<li>It's a little complicated for the targed audience of this post; and<\/li>\n<li>I don't think that most people need it<\/li>\n<\/ul>\n<p>For completeness I'll quickly go over some of the cool, crazy things you can do with containers. You can use tools like <a href=\"https:\/\/kubernetes.io\/\">Kubernetes<\/a> and <a href=\"https:\/\/www.sumologic.com\/glossary\/docker-swarm\/\">Docker Swarm<\/a> with a set of config files to define all your services as Docker containers and how they should all talk to each other. All your containers run somewhere in your Kubernetes\/Swarm cluster, but as a\ndeveloper, you don't really care what server they're on. You just build your Docker containers, write your config file, and push it up to your infrastructure.<\/p>\n<p><img alt=\"maybe kubernetes\" src=\"https:\/\/mattsegal.dev\/django-prod-architecture\/kubernetes-maybe.png\"><\/p>\n<p>Using these \"container orchestration\" tools allows you to decouple your containers from their underlying infrastructure.\nMultiple teams can deploy their apps to the same set of servers without any conflict between their apps.\nThis is the kind of infrastructure that enables teams to deploy <a href=\"https:\/\/www.youtube.com\/watch?v=y8OnoxKotPQ\">microservices<\/a>.\nBig enterprises like Target will have specialised teams dedicated to setting up and maintaining these container orchestration systems, while other teams can use them without having\nto think about the underlying servers. These teams are essentially supplying a \"platform as a service\" (PaaS) to the rest of the organisation.<\/p>\n<p>As you might have noticed, there is probably too much complexity in these container orchestration tools for them to be worth your while as a solo developer or even as a small team.\nIf you're interested in this sort of thing you might like <a href=\"http:\/\/dokku.viewdocs.io\/dokku\/\">Dokku<\/a>, which claims to be \"the smallest PaaS implementation you've ever seen\".<\/p>\n<h2>End of tour<\/h2>\n<p>That's basically everything that I know that I know about how Django can be set up in production.\nIf you're interested in building up your infrastructure skills, then I recommend you try out one of the setups or tools that I've mentioned in this post.\nHopefully I've built up your mental models of how Django gets deployed so that the next time someone mentions \"task broker\" or \"autoscaling\", you have some idea of what they're talking about.<\/p>\n<p>If you enjoyed reading this you might also like other things I've written about <a href=\"https:\/\/mattsegal.dev\/simple-django-deployment.html\">deploying Django as simply as possible<\/a>,\nhow to <a href=\"https:\/\/mattsegal.dev\/offline-tasks.html\">get started with offline tasks<\/a>, how to start <a href=\"https:\/\/mattsegal.dev\/file-logging-django.html\">logging to files<\/a> and <a href=\"https:\/\/mattsegal.dev\/sentry-for-django-error-monitoring.html\">tracking errors<\/a> in prod and my <a href=\"https:\/\/mattsegal.dev\/intro-config-management.html\">introduction to configuration management<\/a>.<\/p>\n<p>If you liked the box diagrams in this post check out <a href=\"https:\/\/excalidraw.com\/\">Exalidraw<\/a>.<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"How to diagnose and fix slow queries with Django Debug Toolbar","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/django-debug-toolbar-performance.html","rel":"alternate"}},"published":"2020-05-09T12:00:00+10:00","updated":"2020-05-09T12:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-05-09:\/django-debug-toolbar-performance.html","summary":"<p>Your Django views are running slowly and you want to make them faster,\nbut you can't figure out what the issue is just by reading the code.\nJust as bad is when you're not sure if you're using the Django ORM correctly - how can you know if the\ncode you \u2026<\/p>","content":"<p>Your Django views are running slowly and you want to make them faster,\nbut you can't figure out what the issue is just by reading the code.\nJust as bad is when you're not sure if you're using the Django ORM correctly - how can you know if the\ncode you write will be slow?<\/p>\n<p>This is where a profiling tool comes in handy.\n<a href=\"https:\/\/django-debug-toolbar.readthedocs.io\">Django Debug Toolbar<\/a> is great\nfor figuring out why your Django views are going slow. This guide will show you how to use DJDT to find and fix slow database queries in your views.<\/p>\n<p>The demo app shown in the video is <a href=\"https:\/\/github.com\/MattSegal\/djdt-perf-demo\">available on GitHub<\/a>.<\/p>\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/9uoI6pvuvYs\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<p>The DJDT docs explain <a href=\"https:\/\/django-debug-toolbar.readthedocs.io\/en\/latest\/installation.html\">how to install<\/a> the toolbar.<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"Simple Django deployment part six: domain setup","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/simple-django-deployment-6.html","rel":"alternate"}},"published":"2020-04-26T18:00:00+10:00","updated":"2020-04-26T18:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-04-26:\/simple-django-deployment-6.html","summary":"<p>We're very nearly done deploying our Django app. There's just one more thing we should take care of.\nHaving a raw IP as our website address is kind of yucky, isn't it?\nYou're not going to ask your friend, boss, or mum to visit 23.231.147.88 to check \u2026<\/p>","content":"<p>We're very nearly done deploying our Django app. There's just one more thing we should take care of.\nHaving a raw IP as our website address is kind of yucky, isn't it?\nYou're not going to ask your friend, boss, or mum to visit 23.231.147.88 to check out your cool new Django app.\nYou want a domain name like mycoolwebsite.xyz! Let's finish up our deployment by setting up a domain for our web app.<\/p>\n<p>Here we will learn how to:<\/p>\n<ul>\n<li>Buy a domain name<\/li>\n<li>Set up a Cloudflare reverse-proxy<\/li>\n<li>Adding our domain name to Django prod settings<\/li>\n<li>Test our setup<\/li>\n<\/ul>\n<p>A quick note before we start - usually you would do this at the start of the process, right after you create your server,\nbecause setting domain name records can take a long time. The reason we're doing it last in this guide is to make sure that you're confident that your app is working before we start fiddling with DNS. If you've never heard of DNS before, I did a short <a href=\"https:\/\/mattsegal.dev\/dns-for-noobs.html\">blog post<\/a> that explains the basics.<\/p>\n<h3>Buy a domain name<\/h3>\n<p>If you already own a domain name for your app your can skip this step.\nTo get a domain name we need to give someone some money.\nWe're going to go to <a href=\"https:\/\/www.namecheap.com\/\">Namecheap<\/a> and buy a domain name. Why Namecheap?\nDomain name registrars exist to sell domains and occasionally fuck you over by raising prices and trying to sell you crap that you don't need. They're generally a pain, so I did a Google search for \"site:reddit.com best domain seller\", and the good people of Reddit seemed to hate Namecheap the least.<\/p>\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/d9XjuXxNPRI\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<h3>Set up Cloudflare<\/h3>\n<p>We're going to use Cloudflare to set up our DNS records. I've written elsewhere on <a href=\"https:\/\/mattsegal.dev\/cloudflare-review.html\">why I like Cloudflare<\/a>. TLDR it's pretty easy to use and provides some nice bonus features like caching your static files, SSL encryption and analytics.<\/p>\n<p>All requests to our domain (mycoolwebsite.xyz) are going to pass through Cloudflare's servers, which are running NGINX under the hood. This kind of set up is called a \"<a href=\"https:\/\/en.wikipedia.org\/wiki\/Reverse_proxy\">reverse proxy<\/a>\", because we have a \"proxy\" (Cloudflare), routing all incoming traffic to our server. This is in contrast to a \"forward proxy\", which deals will outbound traffic.<\/p>\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/GCCBGNKDBIw\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<p>... 30 minutes later ...<\/p>\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/6TWJlVv8Qek\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<h3>Next steps<\/h3>\n<p>Alright! We're done! Congratulations, you've deployed a Django app. Just as a quick recap, you've learned how to:<\/p>\n<ul>\n<li>Use ssh, scp and create SSH keys<\/li>\n<li>Create a cloud virtual machine<\/li>\n<li>Set up your cloud VM<\/li>\n<li>Configure your Django project for deployment<\/li>\n<li>Deploy your Django project to the server<\/li>\n<li>Run your web app using Gunicorn and Supervisor<\/li>\n<li>Set up server logging<\/li>\n<li>Automate the deployment, server setup and database backups<\/li>\n<li>Set up your web app's domain name plus SSL and caching using Cloudflare<\/li>\n<\/ul>\n<p>Now I encourage you to take the things you've learned and write your own Django app and try deploying that.\nIt will probably break at some point, it always does, but I hope you're able to use the skills that you've\npicked up in this guide to debug the problem and fix it.<\/p>\n<p>You've got the basics down, but there is a lot of stuff you can learn about deploying Django and web apps in general.\nSome things you might want to look into at some point:<\/p>\n<ul>\n<li><a href=\"https:\/\/mattsegal.dev\/file-logging-django.html\">Setting up Django logging in production<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/sentry-for-django-error-monitoring.html\">Adding error monitoring<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/offline-tasks.html\">Adding offline tasks<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/simple-scheduled-tasks.html\">Adding offline scheduled tasks<\/a><\/li>\n<li>Start using Git for deployments<\/li>\n<li>Try using Fabric for deployment scripting<\/li>\n<li>Implement \"continuous delivery\" using GitHub actions<\/li>\n<li>Try using PostgreSQL instead of SQLite<\/li>\n<li>Try using NGINX instead of (or in addition to) Cloudflare<\/li>\n<li>Try put your gunicorn server \/ Django app inside of Docker with Docker Swarm<\/li>\n<li>Try out media hosting in AWS S3<\/li>\n<li>Add automated unit tests to your deployment pipeline<\/li>\n<li>Secure your server fail2ban and a firewall<\/li>\n<li>Improve your server setup automation with Ansible<\/li>\n<li>Try a different cloud hosting provider, like AWS or Google Cloud<\/li>\n<\/ul>\n<p>There's an endless list of stuff you can learn, and there's no need to do it all right now,\nbut it's there if you're interested.<\/p>\n<p>If you have any feedback on this guide, or questions about the steps, you can email me at mattdsegal@gmail.com.<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"Simple Django deployment part five: deployment automation","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/simple-django-deployment-5.html","rel":"alternate"}},"published":"2020-04-26T17:00:00+10:00","updated":"2020-04-26T17:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-04-26:\/simple-django-deployment-5.html","summary":"<p>Deploying our Django app involved a lot of different commands, right? It would suck to have to do all that over again, wouldn't it?<\/p>\n<p>Having to manually type all those commands again would be tedious, slow and easy to screw up.\nEven worse, the harder it is to deploy, the \u2026<\/p>","content":"<p>Deploying our Django app involved a lot of different commands, right? It would suck to have to do all that over again, wouldn't it?<\/p>\n<p>Having to manually type all those commands again would be tedious, slow and easy to screw up.\nEven worse, the harder it is to deploy, the less often you are going to do it.\nIf you deployments are infrequent, then they'll contain more features in one big batch and they'll be risker, because there's more things that could go wrong, and it's harder to tell what caused any issues that crop up.\nFrequent, small deployments are key to pumping out lots of valuable code with lower risk.\nThe <a href=\"https:\/\/www.amazon.com.au\/Phoenix-Project-DevOps-Helping-Business\/dp\/0988262592\">Phoenix Project<\/a>\nis a great book that talks more about this idea (srsly give it a read).<\/p>\n<p>So, if we want to deploy fast and often, we're going to need to automate the process. Hell, even if we want to do this again in a week we need to automate the process, because we're definitely going to forget what-the-fuck we just did.\nNo need to get fancy, we can do the whole thing with a bunch of bash scripts.\nYou can get fancy later.<\/p>\n<p>Our goal is that you can run a single bash script and your whole deployment happens.<\/p>\n<p>We'll write these scripts in stages:<\/p>\n<ul>\n<li>Uploading new code to the server<\/li>\n<li>Installing the new code<\/li>\n<li>Single deploy script<\/li>\n<li>Backing up the database<\/li>\n<li>Automating the server setup<\/li>\n<\/ul>\n<h3>Uploading new code to the server<\/h3>\n<p>If you recall, we uploaded code to the server by creating a \"deploy\" directory locally,\nthen uploading that directory to our server. After that we did some clean up work on that directory\nto deal with Python bytecode (pyc) files and Windows line endings.<\/p>\n<p>Let's automate the upload first. The files that we need to copy over are:<\/p>\n<ul>\n<li>requirements.txt for our Python packages<\/li>\n<li>tute for our Django code<\/li>\n<li>scripts for our bash scripts<\/li>\n<li>config for our Gunicorn and Supervisor config<\/li>\n<\/ul>\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/OOYG4ZGOv80\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<h3>Installing the new code<\/h3>\n<p>Now we have automated the process of getting our code onto the server,\nlet's script the bit where we install it in the project dir and run Gunicorn<\/p>\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/R1XDE-NoGAQ\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<p>So, you might have noticed that we stop Gunicorn at the start of the deployment and start it again it at the end. That means your site will be offline during the deployment and if something goes wrong, it'll stay down. You have to log in and manually fix the problem to get it running again.<\/p>\n<p>This is fine for personal projects and low traffic websites - nobody will notice. If you're running some important, high traffic website, then there are techniques to make sure that your website is always running - but we won't go into that here. We're keeping it simple for now.<\/p>\n<h3>Single deploy script<\/h3>\n<p>Alright we're basically done with this section, now all we need to do is combine our two scripts into a master deploy script.<\/p>\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/FnM1fL3-I2E\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<p>That's it, now we can deploy our code over and over in seconds.<\/p>\n<h3>Backing up the database<\/h3>\n<p>This section is optional, it's nice to have, but not a core part of the guide. Skip it if you like.\nHere I'll show you how to back up your database on the server.\nIt's very, very simple to do with SQLite because the database is just a single file.<\/p>\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/Pc6C68RTbfc\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<h3>Automating the server setup<\/h3>\n<p>This section is also optional, it's nice to have, but not a core part of the guide. Skip it if you like.<\/p>\n<p>You will get to a point where you want to move you app to a new server,\nor maybe you've broken your server really badly, or maybe you want to set your server up again slightly differently.\nWhen that time comes, you will not remember how you set this one up: that's why we want to automate our server setup.<\/p>\n<p>Automating your server setup also allows you to do things that were inconceivable before:<\/p>\n<ul>\n<li>run hundreds of servers that are all configured the same way<\/li>\n<li>create a new server for every new deployment (allowing for \"blue-green\" deployments), allowing for zero downtime during deploys<\/li>\n<li>create servers for testing that are identical to your \"live\" production server<\/li>\n<\/ul>\n<p>I talk more about this topic in my video on <a href=\"https:\/\/mattsegal.dev\/intro-config-management.html\">configuration management<\/a>.<\/p>\n<p>So, we want to be able to blow away our server and make a new one with minimal work required. The good news is we're already most of the way there. Our Django app in prod is defined by 3 things at the moment:<\/p>\n<ul>\n<li>our code (we have our code already)<\/li>\n<li>our database (we have automatic backups already)<\/li>\n<li>the server (we know how to set it up, we just need to automate this)<\/li>\n<\/ul>\n<p>Our goal in this section is to run a single script on a new DigitalOcean droplet and it all just works. In addition, we want this script to be \"idempotent\" - this means we want to be able to run it many times on the same server and get (mostly) the same result.<\/p>\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/I4XGu9MXkSE\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<p>This script can get kind of long and hairy, especially as your deployments get more complicated.\nAt some point, you're going to want to use something other than a bash script to automate this process.\nWhen you're ready, I recommend you take a look at <a href=\"https:\/\/github.com\/ansible\/ansible\">Ansible<\/a>,\nwhich is a great tool for writing scripts to automatically setting up servers.\n<a href=\"https:\/\/www.packer.io\/\">Packer<\/a> is also a good tool for using scripts like the one we just wrote to\n\"bake\" a single virtual machine image, which can then be used to instantly create multiple copies of the same virtual machine.<\/p>\n<h3>Next steps<\/h3>\n<p>There's one last thing to do before our website is <em>really<\/em> deployed - <a href=\"https:\/\/mattsegal.dev\/simple-django-deployment-6.html\">give our app a domain name<\/a>.<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"Simple Django deployment part four: run a service","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/simple-django-deployment-4.html","rel":"alternate"}},"published":"2020-04-26T16:00:00+10:00","updated":"2020-04-26T16:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-04-26:\/simple-django-deployment-4.html","summary":"<p>So we've got a problem. Our Django app only runs when we're logged into the server via SSH and running Gunicorn.\nThat's not going to work long term. We need to get Gunicorn running even when we're not around.\nIn addition, if our Gunicorn server crashes because of some bug \u2026<\/p>","content":"<p>So we've got a problem. Our Django app only runs when we're logged into the server via SSH and running Gunicorn.\nThat's not going to work long term. We need to get Gunicorn running even when we're not around.\nIn addition, if our Gunicorn server crashes because of some bug, we want it to automatically restart.<\/p>\n<p>In this section we're going to cover:<\/p>\n<ul>\n<li>Setting up Supervisor<\/li>\n<li>Adding Gunicorn config<\/li>\n<li>Setting up basic logging<\/li>\n<li>Running as root<\/li>\n<\/ul>\n<h3>Setting up Supervisor<\/h3>\n<p>We're going to solve our process supervison problem with <a href=\"http:\/\/supervisord.org\/\">Supervisor<\/a>. It's a program\nthat we can use to run Gunicorn in the background. I chose this tool because a lot of other Django devs use it,\nplus it's pretty easy to install, configure and run.<\/p>\n<p>We can install it into our virtualenv with pip, which is handy:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>pip install supervisor\n<\/code><\/pre><\/div>\n\n<p>Supervisor has several parts that we should know about:<\/p>\n<ul>\n<li><strong>supervisord<\/strong>: the \"<a href=\"https:\/\/en.wikipedia.org\/wiki\/Daemon_(computing)\">daemonized<\/a>\" program that will run Gunicorn as a \"child process\"<\/li>\n<li><strong>supervisorctl<\/strong>: the tool that we will use to send commands to supervisord<\/li>\n<\/ul>\n<p>We'll also be writing some config files to help automate how Supervisor and Gunicorn run<\/p>\n<ul>\n<li><strong>supervisord.conf<\/strong>: a file that we'll need write to configure how supervisord works<\/li>\n<li><strong>gunicorn.conf.py<\/strong>: a file we'll need to write to configure how Gunicorn works<\/li>\n<\/ul>\n<p>Finally, we need to start configuring basic logging. We didn't really need logging before because when we ran \"runserver\" or \"gunicorn\",\nwe could just read the console output on our terminal. We can't do that anymore because we cannot see the terminal. So we need to ask\ngunicorn and supervisord to write their logs to a file somewhere, so we can read them later if we need to. Once we're done, our Django project will look like this when we deploy it:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>\/app\n\u251c\u2500\u2500 env                     Python 3 virtualenv\n\u251c\u2500\u2500 requirements.txt        Python requirements\n\u251c\u2500\u2500 db.sqlite3              Production SQLite database\n\u251c\u2500\u2500 scripts                 Bash scripts\n|   \u2514\u2500\u2500 run-gunicorn.sh     Script to run Gunicorn\n\u251c\u2500\u2500 config                  Config files\n|   \u251c\u2500\u2500 supervisord.conf    Supervisor config\n|   \u2514\u2500\u2500 gunicorn.conf.py    Gunicorn config\n\u251c\u2500\u2500 logs                    Log files\n|   \u251c\u2500\u2500 supervisord.log     Supervisor logs\n|   \u2514\u2500\u2500 gunicorn.access.log Gunicorn access logs\n|   \u2514\u2500\u2500 gunicorn.app.log    Gunicorn application logs\n\u2514\u2500\u2500 tute                    Django project code\n    \u251c\u2500\u2500 tute                Django app code\n    \u251c\u2500\u2500 counter             Django app code\n    \u251c\u2500\u2500 staticfiles         Collected static files\n    \u2514\u2500\u2500 manage.py           Django management script\n<\/code><\/pre><\/div>\n\n<p>It's coming to be a lot of stuff isn't it? When I said this would be a \"simple\" deployment guide, I meant that in a relative sense. \u00af\\_(\u30c4)_\/\u00af<\/p>\n<p>Let's get started by setting up Supervisor to run our Django app using Gunicorn. Unfortunately we can't test this new setup completely on our Windows machine, so we're going to have to upload our files to the server to try this out.<\/p>\n<p>You can find the scripts and config referenced in the video <a href=\"https:\/\/github.com\/MattSegal\/django-deploy\">here<\/a>.<\/p>\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/ny2L15dOf4Q\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<h3>Adding Gunicorn config<\/h3>\n<p>Next we want to tweak how Gunicorn runs a little bit. In particular, we want to set the number of \"workers\". The Gunicorn process runs as a sort of \"master\", which then co-ordinates a bunch of child \"worker\" processes. The <a href=\"https:\/\/docs.gunicorn.org\/en\/stable\/settings.html#workers\">Gunicorn docs<\/a> suggest using 2-4 workers per CPU core (we have 1 on our DigitalOcean VM), but the default is 1.<\/p>\n<p>If we only have 1 worker, and two people send our site a HTTP request, then one of them will need to wait for the other to finish. If we set more workers, it means we can handle more HTTP requests at the same time. Too many workers are kind of pointless because they'll just end up fighting for access to the CPU. So let's pick 3 workers, because we have 1 CPU core, nothing else happening on this machine, and 3 is half way between the recommended 2-4 (which is a very arbitrary way of deciding).<\/p>\n<p>We <em>could<\/em> apply this config change by just adding it as a command line parameter when we run Gunicorn:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>gunicorn tute.wsgi:application --workers <span class=\"m\">3<\/span>\n<\/code><\/pre><\/div>\n\n<p>But this will become unweildy when we configure more and more settings. It's kind of just an aesthetic thing, but I'd rather write this config to a file than as command line parameters. So instead, we can write a <a href=\"https:\/\/docs.gunicorn.org\/en\/stable\/configure.html#configuration-file\">configuration file<\/a> called \"gunicorn.conf.py\" and put all our config in there:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># gunicorn.conf.py<\/span>\n\n<span class=\"n\">bind<\/span> <span class=\"o\">=<\/span> <span class=\"s2\">&quot;0.0.0.0:80&quot;<\/span>\n<span class=\"n\">workers<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">3<\/span>\n<span class=\"c1\"># Add more config here<\/span>\n<\/code><\/pre><\/div>\n\n<p>and then when we run gunicorn we can just do this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>gunicorn tute.wsgi:application -c config\/gunicorn.conf.py\n<\/code><\/pre><\/div>\n\n<p>Let's set up our Gunicorn config.<\/p>\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/KsCJw3skJdQ\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<p>Now that our Gunicorn config has been created, we can set up logging.<\/p>\n<h3>Setting up basic logging<\/h3>\n<p>As I mentioned earlier, we need logging because Gunicorn is now running in the background and we can't see its terminal output.\nThis is important when something goes wrong on in our code and we need to figure out what happened. In this section we'll set up logging so we can see:<\/p>\n<ul>\n<li>what supervisord is doing<\/li>\n<li>what requests Gunicorn is receiving<\/li>\n<li>what Gunicorn is doing, plus Django logs<\/li>\n<\/ul>\n<p>This isn't the <em>perfect<\/em> logging setup, I go into more detail on how we can improve Django logging in production <a href=\"https:\/\/mattsegal.dev\/file-logging-django.html\">in this blog post<\/a>, but it's good enough for now.<\/p>\n<p>When we're done, our logs on the server will look like this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>\/app\n...\n\u2514\u2500\u2500 logs                    Log files\n    \u251c\u2500\u2500 supervisord.log     Supervisor logs\n    \u2514\u2500\u2500 gunicorn.access.log Gunicorn access logs\n    \u2514\u2500\u2500 gunicorn.app.log    Gunicorn application logs\n<\/code><\/pre><\/div>\n\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/ubR--JB5iQM\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<p>Ok we've got logging all set up, looking good! Later on, you might want to also add <a href=\"https:\/\/mattsegal.dev\/sentry-for-django-error-monitoring.html\">error monitoring<\/a> to your app, which alerts you when errors happen.<\/p>\n<h3>Running as root<\/h3>\n<p>Before we move on to automating our deployments, there's an elephant in the room that I'd like to address.\nThis whole time we've been running Gunicorn as the Linux root user.\nIn Windows terminology we'd call this an \"admin\" account.<\/p>\n<p>This setup is a potential security risk. Here's the problem: we've given Gunicorn permission\nto do <em>anything<\/em> to our VM. It can delete all the files, install any programs they want, terminate other processes, whatever.\nThis will be a problem if a hacker figures out how to execute arbitrary code on our Django app, or manipulate our Django app in some other way (like writing to any part of the filesystem).\nAny vulnerability that we accidentally write in our Django app can do maximum damage to our server,\nbecause we've allowed Gunicorn to do everything. The two biggest risks that I see are:<\/p>\n<ul>\n<li>a hacker could trash our server and delete all our shit<\/li>\n<li>a hacker could gain control of our server and use it to mine Bitcoin, <a href=\"https:\/\/www.cloudflare.com\/en-au\/learning\/ddos\/what-is-a-ddos-attack\/\">DDoS<\/a> another server, etc.<\/li>\n<\/ul>\n<p>This is why people say \"don't run Gunicorn as root\", because if you fuck up your code somewhere, or if Gunicorn itself\nis vulnerable somehow, then control of your server and data could be compromised.<\/p>\n<p>So why does this guide have you run Gunicorn as root?<\/p>\n<ul>\n<li>It makes it easier for us to access port 80<\/li>\n<li>It removes some extra work around managing file permissions<\/li>\n<li>It avoids some extra config work around creating new users and assigning user roles<\/li>\n<li>Our server, app and data are all pretty trivial and if they're compromised it's not a big deal<\/li>\n<\/ul>\n<p>As you learn more about deploying web apps and managing infrastructure, you'll need to learn to make your own decisions about\nthe security risks you're willing to take vs. the extra work you'll need to do. For now I think running as root is OK.\nIn the future, especially if you think your app is important, you may want to run Gunicorn as a non-root user and research\nother security measures.<\/p>\n<h3>Next steps<\/h3>\n<p>Now that we've got our Django app up-and-running, all on its own, we can look forward to <a href=\"https:\/\/mattsegal.dev\/simple-django-deployment-5.html\">automating the deployment<\/a>, so we can deploy our code again and again, quickly and easily.<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"Simple Django deployment part three: deploy code","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/simple-django-deployment-3.html","rel":"alternate"}},"published":"2020-04-26T15:00:00+10:00","updated":"2020-04-26T15:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-04-26:\/simple-django-deployment-3.html","summary":"<p>We've got our server set up, and our Django code is ready.\nNow we can actually deploy Django to our server.\nThe goal of this section is to get a basic deployment done.\nWe'll do some automation and introduce some extra tools later.<\/p>\n<p>In this section we'll cover:<\/p>\n<ul>\n<li>Windows line \u2026<\/li><\/ul>","content":"<p>We've got our server set up, and our Django code is ready.\nNow we can actually deploy Django to our server.\nThe goal of this section is to get a basic deployment done.\nWe'll do some automation and introduce some extra tools later.<\/p>\n<p>In this section we'll cover:<\/p>\n<ul>\n<li>Windows line endings<\/li>\n<li>Uploading and running your Django app<\/li>\n<\/ul>\n<h3>Windows line endings<\/h3>\n<p>A quick aside before we start deploying: Windows line endings. These are the curse of every Django developer running Windows.\nThis is one of those technical details that you never want to know about, but they'll bite you in the ass if you ignore them.<\/p>\n<p>The TLDR is that in Linux and MacOS, lines end with the \"\\n\" character.\nOn Windows lines end with \"\\r\\n\", because fuck-you-that's-why.\nThe problem is that your Windows Python files will fail on Linux because they have the wrong line endings.<\/p>\n<p>There are several ways to fix this, including writing our own custom bash or Python scripts to convert these line endings, but for simplicity we'll just use an off-the-shelf tool called <a href=\"https:\/\/linux.die.net\/man\/1\/dos2unix\">dos2unix<\/a>, which I'll show you later.<\/p>\n<p>You can help avoid this problem in VSCode by selecting the \"LF\" option instead of \"CRLF\" for the \"End of Line Sequence\" setting, which is visible in the toolbar on the bottom right hand corner of your screen.<\/p>\n<h1>Uploading and running your Django app<\/h1>\n<p>Let's upload our code to the server and set up our app so we can run it. There are lots of ways to do get your code onto the server: scp, rsync, git. I'm going to stick to using scp to limit the number of new tools needed to do this.<\/p>\n<p>Currently our Django project, on our local machine, looks like this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>\/django-deploy\n\u251c\u2500\u2500 env                     Python 3 virtualenv\n\u251c\u2500\u2500 requirements.txt        Python requirements\n\u251c\u2500\u2500 db.sqlite3              Local SQLite database\n\u2514\u2500\u2500 tute                    Django project code\n    \u251c\u2500\u2500 tute                Django app code\n    \u251c\u2500\u2500 counter             Django app code\n    \u2514\u2500\u2500 manage.py           Django management script\n<\/code><\/pre><\/div>\n\n<p>When we upload our code, we're going to put it in the root user's home directory - \/root\/. It'll look like this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>\/root\n\u2514\u2500\u2500 deploy                  All uploaded code\n    \u251c\u2500\u2500 requirements.txt    Python requirements\n    \u2514\u2500\u2500 tute                Django project code\n        \u251c\u2500\u2500 tute            Django app code\n        \u251c\u2500\u2500 counter         Django app code\n        \u2514\u2500\u2500 manage.py       Django management script\n<\/code><\/pre><\/div>\n\n<p>Then we'll be creating a directory called \/app\/, which will be the final resting place of our code,\nand we will set up our project like this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>\/app\n\u251c\u2500\u2500 env                     Python 3 virtualenv\n\u251c\u2500\u2500 requirements.txt        Python requirements\n\u251c\u2500\u2500 db.sqlite3              Production SQLite database\n\u2514\u2500\u2500 tute                    Django project code\n    \u251c\u2500\u2500 tute                Django app code\n    \u251c\u2500\u2500 counter             Django app code\n    \u251c\u2500\u2500 staticfiles         Collected static files\n    \u2514\u2500\u2500 manage.py           Django management script\n<\/code><\/pre><\/div>\n\n<p>A key idea is that every time we re-deploy our code in the future, we want to delete and re-create the folder \/app\/tute,\nbut we want to keep the database (db.sqlite3), or else we lose all our production data.<\/p>\n<p>What I'm going to show you now is a very manual process, we will automate this later.<\/p>\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/Hm0Dz61_oQ8\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<p>So to recap, the testing we just did looks like this:<\/p>\n<p><img alt=\"gunicorn http\" src=\"https:\/\/mattsegal.dev\/gunicorn-server-http.png\"><\/p>\n<p>We're most of the way there! We've got our Django app running our server.\nThere's just a bit more to go before it's fully deployed.<\/p>\n<h3>Next steps<\/h3>\n<p>To really say that our app is \"deployed\", we need it to run even when we're not around.\nIn the next section, we'll learn how to <a href=\"https:\/\/mattsegal.dev\/simple-django-deployment-4.html\">run Django in the background<\/a><\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"Simple Django deployment part two: local setup","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/simple-django-deployment-2.html","rel":"alternate"}},"published":"2020-04-26T14:00:00+10:00","updated":"2020-04-26T14:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-04-26:\/simple-django-deployment-2.html","summary":"<p>We've got our server set up and ready to host our Django app, now let's focus on preparing our app for deployment.\nThe goal of this section is to set up and test as much of the stuff that we'll be using in production.\nThat way, we can debug issues \u2026<\/p>","content":"<p>We've got our server set up and ready to host our Django app, now let's focus on preparing our app for deployment.\nThe goal of this section is to set up and test as much of the stuff that we'll be using in production.\nThat way, we can debug issues on our computer, instead of on the server.<\/p>\n<p>For this guide I'm going to be creating a Django app from scratch.\nI recommend you follow along and set up your project like I do, rather than trying to deploy an existing Django project.\nYou can try deploy your existing app after you've finished the guide. Remember: new skills on easy terrain.<\/p>\n<p>In this section we'll cover:<\/p>\n<ul>\n<li>Setting up our Python environment<\/li>\n<li>Creating a basic Django app<\/li>\n<li>SQLite limitations<\/li>\n<li>Preparing Django for production<\/li>\n<li>Serving static files in production<\/li>\n<li>Preparing our WSGI server<\/li>\n<li>Windows line endings<\/li>\n<\/ul>\n<h3>Setting up our Python environment<\/h3>\n<p>I assume you've got Python 3 already installed on your computer. If you don't <a href=\"https:\/\/realpython.com\/installing-python\/#windows\">install it now<\/a>.<\/p>\n<p>We're going to be installing some Python packages for our app and we also will want to install the same packages on our server.\nTo keep things consistent, we're going to use a \"virtual environment\" (virtualenv) for this project.\nIn general it's good practice to always use a virtualenv, for these reasons:<\/p>\n<ul>\n<li>It helps maintain consistency between our local project and the deployed project<\/li>\n<li>It helps you keep track of what packages you need to run the project<\/li>\n<li>It helps minimise the number of packages that we need to install when we deploy<\/li>\n<li>It keeps other apps on the same computer from overwriting our packages with different versions<\/li>\n<\/ul>\n<p>Here's how to start our project with a virtualenv.<\/p>\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/8ja20EjR7zs\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<h3>Creating a basic Django app<\/h3>\n<p>Now that we've got Django installed let's create our Django project. This guide covers some of the same ground as the <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/intro\/tutorial01\/\">Django tutorial<\/a>, but we're going to skim through it, because the point isn't to teach you Django basics, it's to teach you how to deploy Django. If you're not familliar with Django then try out the tutorial first.<\/p>\n<p>In addition some of my code (ie. the views) is going to be a little half-assed, since the purpose of the guide is not to show you how to write \"good\" Django views, it's just to get something basic working so we can deploy it.<\/p>\n<p>I've put the <a href=\"https:\/\/github.com\/MattSegal\/django-deploy\">reference code for this guide onto GitHub<\/a>, which you might want to look at while you're following along.<\/p>\n<p>This video will show you how we're going to set up our Django project, and importantly, it will show you how to implement the key features that we want to test later, namely:<\/p>\n<ul>\n<li>A view which interacts with a database model<\/li>\n<li>Some static files (eg. CSS, JS)<\/li>\n<li>Our database setup<\/li>\n<li>The admin panel<\/li>\n<\/ul>\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/fOvQfz8GZeM\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<p>Now we've created our app and it's working locally. The next step is to get it ready for production. Here's a diagram of how we've been running our app and serving requests so far.<\/p>\n<p><img alt=\"runserver http\" src=\"https:\/\/mattsegal.dev\/runserver-http.png\"><\/p>\n<h3 id=\"sqlite\">Is SQLite OK for production?<\/h3>\n\n<p>Before we move on, I want to talk about SQLite quickly. You can skip this bit if you don't care. We'll be using SQLite as our database in development and in production. It'll be two separate databases - we're not going to copy our local SQLite file to the server. The main reason that I'm using SQLite instead of a more advanced database like PostgreSQL or MySQL is because I want to keep this guide as simple as I can.<\/p>\n<p>Is it bad practice to use SQLite in production? Are we taking some shitty shortcut that will bite us in the ass later? Mostly no. Here's what the creators of SQLite <a href=\"https:\/\/www.sqlite.org\/whentouse.html\">have to say<\/a> about running it for webservers:<\/p>\n<blockquote>\n<p>SQLite works great as the database engine for most low to medium traffic websites (which is to say, most websites).<\/p>\n<\/blockquote>\n<p>For our needs, the performance of SQLite is totally fine. There are some limitations to SQLite that are worth mentioning though (<a href=\"https:\/\/djangodeployment.com\/2016\/12\/23\/which-database-should-i-use-on-production\">discussed here<\/a>). One concern is that only one change to the database can <a href=\"https:\/\/www.sqlite.org\/faq.html#q5\">happen at a time<\/a>. Multiple concurrent reads, but only one write:<\/p>\n<blockquote>\n<p>Multiple processes can have the same database open at the same time. Multiple processes can be doing a SELECT at the same time. But only one process can be making changes to the database at any moment in time, however.<\/p>\n<\/blockquote>\n<p>Most website traffic is reads, not writes, so it's not as bad as it sounds.\nStill, what happens in Django when two users try to write to an SQLite database at the same time? I think this will happen:<\/p>\n<ul>\n<li>One user will get a lock on the database, and will write their changes, while the other user will be forced to wait<\/li>\n<li>If the first user finishes quickly enough, then the second user will get their turn - no problem here<\/li>\n<li>If the first user takes too long, then the second user gets an error \"OperationalError: 'database is locked'\"<\/li>\n<\/ul>\n<p>You can <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/ref\/databases\/#database-is-locked-errors\">increase the wait time if you need to<\/a>. I really don't think this is a big issue for low-volume learning projects, or small basic websites with medium traffic.<\/p>\n<p>The other issue worth mentioning is switching from SQLite to another database like PostgreSQL. This probably will be annoying to do, where you need to dump your data to disk as a JSON or something then reload it into Postgres. If this seems like a huge issue for you, then I suggest you follow this guide, then learn how to switch SQLite for Postgres before you fill your database with valuable data. Take small steps.<\/p>\n<p>One thing worth noting is that SQLite is <em>really easy<\/em> to back up. You just make a copy of the file - done!<\/p>\n<h3>Preparing Django for production<\/h3>\n<p>We need to make some changes to our Django settings to prepare our project for production, mostly for security reasons. The big 3 are:<\/p>\n<ul>\n<li><strong>DEBUG<\/strong>: needs to be set to False to prevent Django from leaking information like error messages<\/li>\n<li><strong>SECRET_KEY<\/strong>: needs to be set to something that's actually secret: you can't put it on GitHub<\/li>\n<li><strong>ALLOWED_HOSTS<\/strong>: needs to be set to a whitelist of the IP addresses \/ domain names that your app can use, to prevent cross site request forgery attacks... or something like that<\/li>\n<\/ul>\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/nL6yJOKTzO0\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<p>Our server seems to be <em>mostly<\/em> working with our new production settings...\nother than our static files mysteriously breaking. Let's fix that next.<\/p>\n<h3>Serving static files in production<\/h3>\n<p>So your static files (CSS, JS, images, etc) work fine when DEBUG=True, but they're broken when DEBUG=False.\nThis doesn't seem like a \"debug\" thing... what the fuck? Right?\nThey were working before!?!? Whose ideas was this?<\/p>\n<p>Aren't you glad you found out about this problem <em>before<\/em> you tried to deploy your app?<\/p>\n<p>Many Django developers have been slapped in the face by this surprise.\nIf you want to go outside and scream now's a good time.<\/p>\n<blockquote>\n<p>AIIIIIIIIIIIIIIEEEEAAAAAAAAAAAAHHHH!!!!<\/p>\n<\/blockquote>\n<p>Computers can be frustrating! I like Django and the people who built it.\nThat said, this is one of the few times where I feel like the framework lets you down.\nDjango's docs on the <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/howto\/static-files\/deployment\/\">subject of deploying static files<\/a> are somewhere between cryptic and infuriating.\nThey're usually good docs too!<\/p>\n<p>The reason that the static files break when DEBUG=False is that there are lots of different ways to serve static content.\nWhen you are in DEBUG=True mode, Django helpfully serves your static files for you.\nWhen you set DEBUG=False, you're on your own - Django forces you to figure out how you're going to serve static files in production.<\/p>\n<p>There are several options available: most of the choices that are made around hosting costs, the other tech tools you're using\nbandwidth, performance - shit we don't care about right now.\nWe want the simplest solution for serving static files in production.<\/p>\n<p>As far as I know <a href=\"http:\/\/whitenoise.evans.io\/en\/stable\/\">Whitenoise<\/a> is the simplest way to serve static files in production:<\/p>\n<blockquote>\n<p>Radically simplified static file serving for Python web apps... None of this is rocket science, but it\u2019s fiddly and annoying and WhiteNoise takes care of all it for you.<\/p>\n<\/blockquote>\n<p>Sounds good right? It basically just does what runserver was doing before we set DEBUG=False, except maybe a bit better, or something. Their <a href=\"http:\/\/whitenoise.evans.io\/en\/stable\/index.html\">documentation<\/a> and <a href=\"http:\/\/whitenoise.evans.io\/en\/stable\/index.html#infrequently-asked-questions\">FAQ<\/a> goes over what it does for you. We're going to use the CloudFlare CDN in a later part of this guide to cache our static files, so that will solve most of our performance concerns.<\/p>\n<p>Let's follow their <a href=\"http:\/\/whitenoise.evans.io\/en\/stable\/django.html\">guide<\/a> and set up Django to use Whitenoise for static files. Before we get to the video let's go over the important bits.<\/p>\n<p>First we have to install it<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>pip install whitenoise\n<\/code><\/pre><\/div>\n\n<p>We also have to set STATIC_ROOT in our Django settings. STATIC_ROOT is a folder where Django will dump all of your static files when you run the \"collectstatic\" management command. Whitenoise looks inside this folder when DEBUG=False, so it's important we set it, and run \"collectstatic\" when we deploy. We'll go over this more in the video.<\/p>\n<p>Alright, let's set up Whitenoise and solve our static files problem.<\/p>\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/97UQM-Cfhxs\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<h3 id=\"wsgi\">Preparing our WSGI server<\/h3>\n\n<p>So far we've been using the \"runserver\" management command to run our Django code and serve HTTP requests.\nIt works pretty well for development - the way it auto restarts when files change is pretty handy.\nThere's some trouble with running runserver in production though -the Django docs <a href=\"https:\/\/docs.djangoproject.com\/en\/2.2\/ref\/django-admin\/#runserver\">say it best<\/a>:<\/p>\n<blockquote>\n<p>DO NOT USE THIS SERVER IN A PRODUCTION SETTING. It has not gone through security audits or performance tests. (And that\u2019s how it\u2019s gonna stay. We\u2019re in the business of making Web frameworks, not Web servers, so improving this server to be able to handle a production environment is outside the scope of Django.)<\/p>\n<\/blockquote>\n<p>Why <em>exactly<\/em> is using runserver in prod a bad idea? Honestly I don't know, I've never tried. Something about security and performance... here's the thing: when the people writing the software tell you not to use it production (in all caps no less), it's best to just listen to them, unless you're confident that you understand the risks and benefits.<\/p>\n<p>So... what do we use to run our Django app instead? We're going to use <a href=\"https:\/\/gunicorn.org\/\">Gunicorn<\/a>, basically because it's a popular WSGI server and I'm familliar with it and it seems OK. Another widely used contender is <a href=\"https:\/\/uwsgi-docs.readthedocs.io\/en\/latest\/\">uWSGI<\/a>. I've seen <a href=\"http:\/\/docs.pylonsproject.org\/projects\/waitress\/en\/stable\/\">Waitress<\/a> recommended for running on Windows, but I've never tried it myself.<\/p>\n<p>You might be wondering what \"<a href=\"https:\/\/wsgi.readthedocs.io\/en\/latest\/what.html\">WSGI<\/a>\" (\"Web Server Gateway Interface\") means. WSGI is a type of \"interface\". I think it's much easier to explain with examples than to get too theoretical.<\/p>\n<p>Here are some WSGI compatible web frameworks:<\/p>\n<ul>\n<li>Django<\/li>\n<li>Flask<\/li>\n<li>Pyramid<\/li>\n<li>web2py<\/li>\n<\/ul>\n<p>Here are some WSGI compatible web servers:<\/p>\n<ul>\n<li>Gunicorn<\/li>\n<li>uWSGI<\/li>\n<li>CherryPy<\/li>\n<li>Apache's mod_wsgi module<\/li>\n<\/ul>\n<p>Web frameworks (eg. Django) are just some Python code, you need a web server to actually run the code and translate incoming HTTP requests (which are just text) into Python objects. The WSGI specification makes it so that any WSGI compatible webserver can run any WSGI compatible web framework, which means:<\/p>\n<ul>\n<li>Gunicorn can run Django<\/li>\n<li>Gunicorn can run Flask<\/li>\n<li>CherryPy can run web2py<\/li>\n<li>mod_wsgi can run Django<\/li>\n<li>... etc etc etc ...<\/li>\n<\/ul>\n<p>This is a good thing because it means that if you are using a particular web framework (eg. Django), you have a lot of choices for which web server you run (eg. Gunicorn). It's also good for web server developers, because lots of people with different web frameworks can use their tools.<\/p>\n<p>With that out of the way, let's get stuck into using Gunicorn instead of runserver to run our Django app.<\/p>\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/wHmpB2AEmZY\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<p>So before we were doing this:<\/p>\n<p><img alt=\"runserver http\" src=\"https:\/\/mattsegal.dev\/runserver-http.png\"><\/p>\n<p>Now we're doing this (hypothetically if Gunicorn actually worked on Windows):<\/p>\n<p><img alt=\"gunicorn http\" src=\"https:\/\/mattsegal.dev\/gunicorn-http.png\"><\/p>\n<p>Nothing too crazy.<\/p>\n<h3>Next steps<\/h3>\n<p>Now that we've done our local setup, we're ready to <a href=\"https:\/\/mattsegal.dev\/simple-django-deployment-3.html\">deploy Django to the server<\/a><\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"Simple Django deployment part one: infrastructure","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/simple-django-deployment-1.html","rel":"alternate"}},"published":"2020-04-26T13:00:00+10:00","updated":"2020-04-26T13:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-04-26:\/simple-django-deployment-1.html","summary":"<p>In order to deploy our Django app, we need a somewhere to run it: we need a server.\nIn this section we'll be setting up our server in \"the cloud\".\nDoing this can be fiddly and annoying, especially if you're new, so we want to get it right first before \u2026<\/p>","content":"<p>In order to deploy our Django app, we need a somewhere to run it: we need a server.\nIn this section we'll be setting up our server in \"the cloud\".\nDoing this can be fiddly and annoying, especially if you're new, so we want to get it right first before we involve our Django app. By the end of this section we will learn how to:<\/p>\n<ul>\n<li>Install ssh and scp on Windows<\/li>\n<li>Create a SSH key<\/li>\n<li>Set up our cloud web server<\/li>\n<li>Learn how to access our server, upload files<\/li>\n<li>Test our setup<\/li>\n<\/ul>\n<h3>Installing our tools<\/h3>\n<p>We'll need some tools to access the server:<\/p>\n<ul>\n<li>the \"bash\" <a href=\"https:\/\/en.wikipedia.org\/wiki\/Unix_shell\">shell<\/a> for scripting<\/li>\n<li>the \"ssh\" tool for logging into our web server<\/li>\n<li>the \"scp\" tool to transfer file to our server<\/li>\n<\/ul>\n<p>We need to install these tools on Windows and the fastest and easiest way I know of is to just <a href=\"https:\/\/git-scm.com\/download\/win\">install Git<\/a>.\nWe won't be using Git, just some of the tools that get installed with it. You can <a href=\"https:\/\/www.codecademy.com\/learn\/learn-git\">learn Git<\/a> some other time.\nIf you're using a Mac or Linux you can skip this step and open up a terminal window.<\/p>\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/yizAaMHUC5w\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<h3>Creating a SSH key<\/h3>\n<p>We'll be using an \"SSH key\" a way to authenticate ourselves when we log in to the server with ssh.\nWe need this key for when we create our server, so we're doing this bit first.\nIt's possible to just use a username \/ password when logging in via SSH, but creating a key is more convenient in the long run.\nIn particular, using a key means you don't have to type in a password every time you want to access the server.<\/p>\n<p>In this video, we'll be creating an SSH key using the \"ssh-keygen\" command in bash:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>ssh-keygen -C <span class=\"s2\">&quot;mattdsegal@gmail.com&quot;<\/span>\n<\/code><\/pre><\/div>\n\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/BIc1TWrVQcw\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<p>Now that we've created our key, we're ready to use it to log into web servers. To recap:\nthe SSH key we just created has two parts, a public key (id_rsa.pub) and a private key (id_rsa).\nBoth keys will be stored in ~\/.ssh by convention. You can read your public key like this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>cat ~\/.ssh\/id_rsa.pub\n<\/code><\/pre><\/div>\n\n<p>A public key is like your \"username\" when logging in - it's public information and it's OK to share it. It looks like this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVtWYN+uDbdG6RtmU8vcaj1cxYM0kK6565LFa\nMMkolZZlSA6MhfWfGUmIGswIHJ\/yjQCRQRihlEdm0VQJgsFBtK36J\/U2\/u+cMjGXwN\/9swYBsnj\n8bSMRzYc2s6PeshYmefpD80dWsvW550zqHmOwnKxeiwpz1q+rqUgT\/xd0nOATw92nx5CS7ozhnL\nt0FA0r0fk9LGih473Ho4\/22fsAGXTcnMV5VoDgeBP4z8BLt16pKD8fgSGB8OG3\/bN6udY54TcM2\nrFjfN8yP+Vcbs5xBd3HaTu8Z42IPdC46Z25WMt285FLLZyUqWY36CrQZoTEf9F6aCkFgwtOCN81\nu0Qr1 foo@bar.com\n<\/code><\/pre><\/div>\n\n<p>It's usually all just long one line of text.\nThe \"ssh-rsa\" part means it's a key for SSH and uses RSA encryption.\nThe \"AAAAB3N...u0Qr1\" part is your actual key, it uniquely identifies your keypair.\nThe \"foo@bar.com\" part is just a comment, it can be anything, but by convention it's your email.<\/p>\n<p>A private key is like your password - do not share it with anyone. Private keys look like this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEAlbVmDfrg23RukbZlPL3Go9XMWDNJCuueuSxWjDJKJWWZUgOj\nIX1nxlJiBrMCByf8o0AkUEYoZRHZtFUCYLBQbSt+if1Nv7vnDIxl8Df\/bMGAbJ4\/\nG0jEc2HNrOj3rIWJnn6Q\/NHVrL1uedM6h5jsJysXosKc9avq6lIE\/8XdJzgE8Pdp\n8eQku6M4Zy7dBQNK9H5PSxooeO9x6OP9tn7ABl03JzFeVaA4HgT+M\/AS7deqSg\/H\n4EhgfDht\/2zernWOeE3DNqxY3zfMj\/lXG7OcQXdx2k7vGeNiD3QuOmduVjLdvORS\ny2clKlmN+gq0GaExH\/RemgpBYMLTgjfNbtEK9QIDAQABAoIBAQCAtWwAKOiYxAkr\njTyMdDwLLwx359+sW9YiLVRbRAErFaYzNJ1TdZV6k+ljCRN9Q4uYbtTJjwe7nRUm\nTM+2gN8kfHhV+kiVxt5lk28wj3Qx9EqNF5\/5vR3odPV26vPEhypB8V6FfYHO+S25\n3zg6y+Z75jhz3g1DyYI14j4aB+qSg+5YSQ56vG+vhutYD41XVp3bkgD76kL8QFxd\nq2cTpF5WSoZF49CaPhE2PnHoMZibLRQUOG+wJWkrQHcU+UnoWQvQUkqGNeRYpmI5\n49Umk3b+\/MQr0Dj6vuT2ZKqgFjr2FEu8AoA0tCZu8GUXWS1LshuZuqb1D0ZY+KOj\n1SpEY48hAoGBAMXAbA8iNeON93yhOf7GqRCecL\/wuyn6KqifCtEg8HbhbafcWBsd\n\/prEnEJHGRTU+omfLx7uVJmAgkVhj\/uHl8K9L2qvlqkbBqF\/rWPwWBi94G\/cszn4\ntYb91sTnnOk+QRjs\/bGSCcv6kGlv2Bv0YYie0K6oQNPD9SqXCAit5hCDAoGBAMHO\nQv2+JcrrfLjoCbUaPaJ2sblO6Bq1RpUsltE1lL8nldTu\/MgUIA3dEExg\/hl7YLYW\nvQVgNZkJQxHf1UJgZF\/FTfw8yYaN4CKX4lOXDeKW8kN8xTtQQCPi77w\/jxz5DUau\nwmhqYOVwrCeqvLI4qUoEp6oOsQ83GxjHKXPfNK0nAoGAQD5aHLSFg068xz1tpOqP\nRDnk8UZY17NRJoS8s+IanNRxlmYMLYsaCtey2AlXCaCDYDBZ05ej3laUe8vNRe7w\nC7EAdY1jyb5g8hiTkPMk+6y7\/Dtb8optFtTicAe6vz+dUGa1qHmEO0NEpSxTrgk\/\nom3N59\/7Z5Cy1kpIruEn69cCgYBgRYCboVgOq9nB1Gn2D3nseT+hiKPdmIzeT07\/\nz7j7F8PjCXCCRxUBLf4Joui2acZJzZPJ1tfpFGO\/vkumdFGIDW\/Gy79j2pgrNv2T\nfmbEVy0y\/wjOhPfHm9Rw07XYs5K3uNoTmjxV3Rl3fuXLNkBJ53QOEsw7fak1LsHV\nsFvvYwKBgDTRNKz3217jYjuAzeluHQHsfrUKn73DzckivDHr35m1JR37rlNXGWRE\nJlN82KNevAqrDabwYPZnkDrPMGpLQi1A1icUtiBXRRskeR6ULxOASVyAJ3N1WV8T\n1\/g0+hahPeNFGQG649Z1d5WYSJeVbz7is3MiVGYaQu+iNz9VXNq5\n-----END RSA PRIVATE KEY-----\n<\/code><\/pre><\/div>\n\n<p>... I'm not actually using this one so it's OK to share.<\/p>\n<h3>Creating the server<\/h3>\n<p>You've probably heard the word \"server\" used to refer to a dozen different things, so let me be specific.\nOur server will be a Linux virtual machine (VM), which we are going to rent from DigitalOcean, a cloud hosting company.\nDigitalOcean will run our VM in one of their datacenters, which is a <a href=\"https:\/\/lh3.googleusercontent.com\/7D8_SzSQQn-uDeKq4R7SSER5LO7fjsnkCLJ-uZG443cKHFS20nU-SyvlzXaGP97Fgt31MYJdgy94563uETi9jbosUMYQzO95-H0PRg=w2114-h1058-n\">big building<\/a> that is <a href=\"https:\/\/www.pon-cat.com\/application\/files\/6215\/3995\/7501\/Datacenter-Pon_Power.jpg\">full of computers<\/a>. For our purposes, this VM is a stand-alone computer that is for our private usage, with a static IP address which we can use to find it online.<\/p>\n<p>The first thing you need to do is create an account with <a href=\"https:\/\/www.digitalocean.com\/\">DigitalOcean<\/a>. The only reason I've chosen this company is because they have a nice web user interface and I already use them. Other than that, there's no reason you couldn't also use Linode, AWS, Google Cloud or Azure to do the exact same thing. They all provide Linux web servers for rent.<\/p>\n<p>Once you've created your account, you can follow this video for the rest of the setup.\nI'm not sure exactly when they're going to ask you to put your credit card details it, but have a credit card ready.<\/p>\n<p>By the end of this video we'll have created our server and we'll have an IP address, which we can use to log into the server.<\/p>\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/mdRTN-rzi94\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<h3>Setting up the server<\/h3>\n<p>The main tools that we'll need to run our Django app on this server are Python 3 and pip, the Python package manager.\nYou'll find that Python 3 is already installed on our Ubuntu server, but we need to install pip.\nWe'll be using the <a href=\"https:\/\/devconnected.com\/apt-package-manager-on-linux-explained\/\">apt package manager<\/a> to download and install pip.<\/p>\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/wHbOsG1UV9Q\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<p>By the way, it turns out that \"LTS\" stands for <a href=\"https:\/\/en.wikipedia.org\/wiki\/Long-term_support\">Long Term Support<\/a> and refers to Ubuntu's policy of how they plan to provide patches in the future (not super relevant to this guide).<\/p>\n<h3>Uploading files and troubleshooting HTTP<\/h3>\n<p>So now we know how to create a server, log in with ssh and install the software we need to run Django.\nNext I will show you how to upload files to the server with scp.\nIn addition, I'll show you how to run a quick and easy HTTP web server, which can be useful for debugging later. You will need <a href=\"https:\/\/realpython.com\/installing-python\/#windows\">Python 3 installed<\/a> on your desktop for this step.<\/p>\n<div class=\"yt-embed\">\n    <iframe \n        src=\"https:\/\/www.youtube.com\/embed\/sQNNsetMZfg\" \n        frameborder=\"0\" \n        allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" \n        allowfullscreen\n    >\n    <\/iframe>\n<\/div>\n\n<p>If you want to take a 40 minute side-quest I recommend checking out Brian Will's \"The Internet\" videos to learn more about what HTTP, TCP, and ports are: <a href=\"https:\/\/www.youtube.com\/watch?v=DTQV7_HwF58\">part 1<\/a>, <a href=\"https:\/\/www.youtube.com\/watch?v=3fvUc2Dzr04&amp;t=167s\">part 2<\/a>, <a href=\"https:\/\/www.youtube.com\/watch?v=_55PyDw0lGU\">part 3<\/a>, <a href=\"https:\/\/www.youtube.com\/watch?v=yz3lkSqioyU\">part 4<\/a>.<\/p>\n<p>I'll recap on the HTTP troubleshooting, so you understand why we just did that.\nThe main purpose was to get us ready to troubleshoot our Django app later.<\/p>\n<p>We ran a program (http.server) on our Linux VM using Python.\nThat program was listening for HTTP requests on port 80, and when someone sends it a GET request, it responds with the contents of the index.html file that we wrote and uploaded.<\/p>\n<p>If we use our web browser and visit our web server's IP address (64.225.23.131 in my case), our web browser will send an HTTP GET request to our server on port 80. Sometimes this works and gets the the HTML, but under some circumstances it will fail. The point of our troubleshooting is to figure out why it is failing. It could be that:<\/p>\n<ul>\n<li>Our computer is not connected to the internet<\/li>\n<li>Our web server is turned off<\/li>\n<li>We cannot access our web server over the internet<\/li>\n<li>A firewall on our web server is blocking port 80<\/li>\n<li>etc. etc. etc<\/li>\n<\/ul>\n<p>One way we can figure out what's going on is to log into the VM using ssh and make a HTTP GET request using curl. If the curl request works, then we know that the our HTTP server program is working and serving requests <em>inside<\/em> the VM, and the problem is something between our computer and the VM. Once we've narrowed down the exact problem, then we can figure out how to fix it.<\/p>\n<p><img alt=\"troubleshooting http\" src=\"https:\/\/mattsegal.dev\/troubleshoot-http.png\"><\/p>\n<p>This style of troubleshooting will become useful when we start setting up our Django app on the server.<\/p>\n<h3>Next steps<\/h3>\n<p>Now our server is ready to serve Django and we know how to troubleshoot HTTP connections.\nNext we will <a href=\"https:\/\/mattsegal.dev\/simple-django-deployment-2.html\">prepare and test Django locally<\/a>.<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"Simple Django deployment: a guide","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/simple-django-deployment.html","rel":"alternate"}},"published":"2020-04-26T12:00:00+10:00","updated":"2020-04-26T12:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-04-26:\/simple-django-deployment.html","summary":"<p>You're learning web development with Django. You've followed the <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/intro\/tutorial01\/\">official introductory tutorial<\/a> and you can get a Django app working on your local computer. Now you want to put your web app onto the internet. Maybe it's to show your friends, or you actually want to use it for something \u2026<\/p>","content":"<p>You're learning web development with Django. You've followed the <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/intro\/tutorial01\/\">official introductory tutorial<\/a> and you can get a Django app working on your local computer. Now you want to put your web app onto the internet. Maybe it's to show your friends, or you actually want to use it for something, or maybe you just want to learn how to deploy Django apps. This guide has five parts:<\/p>\n<ol>\n<li><a href=\"https:\/\/mattsegal.dev\/simple-django-deployment-1.html\">Server setup<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/simple-django-deployment-2.html\">Prepare and test Django locally<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/simple-django-deployment-3.html\">Deploy Django to the server<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/simple-django-deployment-4.html\">Run Django in the background<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/simple-django-deployment-5.html\">Automate the re-deployment<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/simple-django-deployment-6.html\">Domain setup<\/a><\/li>\n<\/ol>\n<p>You can start the guide with part 1 now. If you're interested, read on to learn more about what motivated me to write this.<\/p>\n<h3>Stuck, frustrated, confused<\/h3>\n<p>You've probably tried <a href=\"https:\/\/www.digitalocean.com\/community\/tutorials\/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-18-04\">tutorials like this<\/a> which give you a bunch of steps to follow, commands to type, files to configure. This is how I learned to deploy Python web apps: with online tutorials and a lot of Googling. When you follow these guides, you have no fucking idea what you're actually doing. Why do you use that tool? Why do you type that command? You may as well be <a href=\"https:\/\/youtu.be\/nAQBzjE-kvI?t=33\">learning magic at Hogwarts<\/a>. You could easily swap:<\/p>\n<blockquote>\n<p>What is apt? Why am I using it to install postgresql-contrib and libpq-dev?<\/p>\n<\/blockquote>\n<p>with<\/p>\n<blockquote>\n<p>Why do I have to say Wingardium Levios<em>aaa<\/em> not Leviosa<em>rrr<\/em> to get my spell to work?<\/p>\n<\/blockquote>\n<p>It's not your fault. These kinds of guide throw a lot of unfamilliar tools and concepts at you without taking the time to teach you about them. The DigitalOcean guide above smacks you with:<\/p>\n<ul>\n<li>apt package manager<\/li>\n<li>PostgreSQL installation<\/li>\n<li>PostgreSQL database admin<\/li>\n<li>Python virtual environments<\/li>\n<li>Prod Django settings<\/li>\n<li>Running a gunicorn WSGI server<\/li>\n<li>Firewall configurations<\/li>\n<li>Systemd configuration<\/li>\n<li>Socket file setup<\/li>\n<li>NGINX reverse proxy setup<\/li>\n<\/ul>\n<p>It also requires that you know:<\/p>\n<ul>\n<li>How to spin up a new a web server<\/li>\n<li>How to login via SSH<\/li>\n<li>How to set DNS records<\/li>\n<li>How to get your Django code onto the server<\/li>\n<\/ul>\n<p>Some of these tools and skills are necessary, some of them are not. If you don't follow their instructions perfectly then you can get stuck and have no idea how to get unstuck. Then you get frustrated, discouraged and embarrassed that you suck so much at deployment. It's pretty common for new developers to struggle for days, even weeks to get their first web app deployed.<\/p>\n<p>Hitting a wall when trying to deploy your Django app isn't inevitable. I used to work as a ski instructor (software pays better) and I was taught a saying:<\/p>\n<blockquote>\n<p>Teach new skills on easy terrain. On hard terrain, stick to the old skills.<\/p>\n<\/blockquote>\n<p>This means that you shouldn't try teaching a fancy new technique on the steepest, hardest runs.\nDeploying web applications is <em>hard<\/em>. It gets easier with time, but it's got a nasty learning curve. It's easier to learn if we minimise the number of new skills and try to keep you in a familiar environment.<\/p>\n<h3>Minimal new tools, small steps<\/h3>\n<p>That's the focus of this guide. I want to help you achieve lots of small, incremental wins where you gain one small skill, then another, until you have all the skills you need to deploy your Django app. I want to you to understand what the fuck is going on so you don't get stuck. I want to introduce as few new tools as possible.<\/p>\n<p>Here are the new technologies that I propose we learn to use:<\/p>\n<ul>\n<li>A Linux virtual machine in the cloud for hosting (DigitalOcean)<\/li>\n<li>SSH and SCP for accesing the server<\/li>\n<li>git-bash shell scripting<\/li>\n<li>Python virtual environments<\/li>\n<li>Gunicorn WSGI server for running your app<\/li>\n<li>Supervisor for keeping Gunicorn running<\/li>\n<li>Whitenoise Python library to serve static files<\/li>\n<li>Cloudflare SaaS tool for DNS, static file caching, SSL<\/li>\n<\/ul>\n<p>That's still a lot of tools, despite trying to keep it small an simple. Here are some things we will not be using:<\/p>\n<ul>\n<li>PostgreSQL database<\/li>\n<li>NGINX reverse proxy<\/li>\n<li>Containers (eg. Docker, Kubernetes)<\/li>\n<li>Config management tools (eg. Ansible, Fabric)<\/li>\n<li>Git version control<\/li>\n<\/ul>\n<p>You should give them a try sometime... just not yet.<\/p>\n<blockquote>\n<p>But don't professional web developers use NGINX\/Docker\/Postgres\/etc? That's what people on Reddit say! I don't want to learn bad practices :(<\/p>\n<\/blockquote>\n<p>It's true that these are all great tools. I use them often, but I think they will make learning to deploy Django unnecessarily complicated.\nThe good news is that you can always add them to your infrastructure later on.\nOnce you've got this simple deployment down then you can mix it up: you can add NGINX, Postgres and Docker if you like.<\/p>\n<h3>The guide<\/h3>\n<p>I am going to assume that you are using Windows for the guide, partly because it's what most new developers use, and partly because it's the worst-case scenario.\nThat's right: doing this stuff on Windows is hard-mode.\nIf you have a Mac or Linux desktop, then you can still follow along - there will just be slightly fewer things for you to do.<\/p>\n<p>Also, just so you know, this guide will involve buying a domain name ($2 - $10 USD \/ year) and using a paid cloud service (5 bucks \/ month).\nIf you're not willing (or unable) to get your credit card out and pay for some stuff, then you will not be able to complete every step.<\/p>\n<p>I said this was a \"simple\" guide, but I didn't say it's short: it's surprisingly long in fact. This guide has five steps, which I suggest you do in order:<\/p>\n<ol>\n<li><a href=\"https:\/\/mattsegal.dev\/simple-django-deployment-1.html\">Server setup<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/simple-django-deployment-2.html\">Prepare and test Django locally<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/simple-django-deployment-3.html\">Deploy Django to the server<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/simple-django-deployment-4.html\">Run Django in the background<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/simple-django-deployment-5.html\">Automate the re-deployment<\/a><\/li>\n<li><a href=\"https:\/\/mattsegal.dev\/simple-django-deployment-6.html\">Domain setup<\/a><\/li>\n<\/ol>","category":{"@attributes":{"term":"Django"}}},{"title":"4 tips for debugging in Django","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/django-debug-tips.html","rel":"alternate"}},"published":"2020-04-12T12:00:00+10:00","updated":"2020-04-12T12:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-04-12:\/django-debug-tips.html","summary":"<p>You've got a bug in your Django code and you can't quite figure out what's wrong. You know there's a problem, but you can't quite pin down where it's coming from. This post will share 4 tips which will help you speed up your bug catching.<\/p>\n<h3>Dig deeper in your \u2026<\/h3>","content":"<p>You've got a bug in your Django code and you can't quite figure out what's wrong. You know there's a problem, but you can't quite pin down where it's coming from. This post will share 4 tips which will help you speed up your bug catching.<\/p>\n<h3>Dig deeper in your print statements<\/h3>\n<p>Using <code>print<\/code> to view data is the most basic debugging method. You're probably already doing this, so I'm going to show you how to squeeze more info out of your objects when you print them.<\/p>\n<p>Basic print usage for debugging looks like this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># views.py<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">my_view<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">thing<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Things<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span><span class=\"o\">.<\/span><span class=\"n\">last<\/span><span class=\"p\">()<\/span>\n    <span class=\"nb\">print<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;Check thing:&quot;<\/span><span class=\"p\">,<\/span> <span class=\"n\">thing<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">HttpResponse<\/span><span class=\"p\">(<\/span><span class=\"sa\">f<\/span><span class=\"s2\">&quot;The thing is called <\/span><span class=\"si\">{<\/span><span class=\"n\">thing<\/span><span class=\"o\">.<\/span><span class=\"n\">name<\/span><span class=\"si\">}<\/span><span class=\"s2\">&quot;<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n<p>The problem is that when you're looking at Python objects, you might only see a string representing the object, rather than the data you want. For example the above code will print this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>Check thing: &lt;Thing: 1&gt;\n<\/code><\/pre><\/div>\n\n<p>This is not helpful for our debugging, but there's a better way. We can use <code>pprint<\/code>, which \"pretty prints\" dictionaries, and the <code>__dict__<\/code> attribute, which is present on every Python object, to dig into the data in more detail:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># views.py<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">my_view<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">thing<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Things<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span><span class=\"o\">.<\/span><span class=\"n\">last<\/span><span class=\"p\">()<\/span>\n    <span class=\"kn\">from<\/span> <span class=\"nn\">pprint<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">pprint<\/span>\n    <span class=\"n\">pprint<\/span><span class=\"p\">(<\/span><span class=\"n\">thing<\/span><span class=\"o\">.<\/span><span class=\"vm\">__dict__<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">HttpResponse<\/span><span class=\"p\">(<\/span><span class=\"sa\">f<\/span><span class=\"s2\">&quot;The thing is called <\/span><span class=\"si\">{<\/span><span class=\"n\">thing<\/span><span class=\"o\">.<\/span><span class=\"n\">name<\/span><span class=\"si\">}<\/span><span class=\"s2\">&quot;<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n<p>With this method we will see a nicely formatted dict, showing all the data attached to the <code>thing<\/code> object:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>{\n    &quot;_state&quot;: &lt;django.db.models. ...&gt;,\n    &quot;id&quot;: 1,\n    &quot;name&quot;: &quot;the thing&quot;,\n    &quot;weight&quot;: 12,\n}\n<\/code><\/pre><\/div>\n\n<p>Now you can dig deeper into your objects when printing.<\/p>\n<p>Leaving a bunch of print statements in your code can pollute your app's console output. You can keep the printing but reduce the noise by <a href=\"https:\/\/mattsegal.dev\/file-logging-django.html\">setting up logging<\/a>, which then enables you to toggle how noisy your logs are using <a href=\"https:\/\/docs.python.org\/3\/howto\/logging.html\">log levels<\/a>.<\/p>\n<h3>Python's built-in debugger<\/h3>\n<p>Finding bugs via print works, but it can be a slow and tedious process. You might have to run the same code dozens of times to find the problem. Wouldn't it be nice to just stop the code on a particular line and then check a bunch of variables? You can do this with Python's built-in debugger. You can get started with it by following <a href=\"https:\/\/mattsegal.dev\/django-debug-pdb.html\">this guide on using pdb<\/a>.<\/p>\n<h3>Check your insanity with assertions<\/h3>\n<p>At some point during debugging you start to question your sanity - you don't know what to believe anymore. You start to question everything you've ever known about programming.<\/p>\n<blockquote>\n<p>When debugging, you must first accept that something you believe is true is not true. If everything you believed about this system were true, it would work. It doesn't, so you're wrong about something. (<a href=\"https:\/\/twitter.com\/cocoaphony\/status\/1224364439429881856\">source<\/a>)<\/p>\n<\/blockquote>\n<p>Using Python's <code>assert<\/code> statement is a quick and easy way to check if something that you believe is true, is <em>acutally true<\/em>. <code>assert<\/code> is pretty simple:<\/p>\n<ul>\n<li>You whack <code>assert<\/code> in your code with an expression<\/li>\n<li>If the expression is truthy then nothing happens<\/li>\n<li>If the expression is falsy then <code>assert<\/code> throws an <code>AssertionError<\/code><\/li>\n<\/ul>\n<p>Simple, but quite useful. Here are some quick examples:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># All OK, nothing happens<\/span>\n<span class=\"k\">assert<\/span> <span class=\"kc\">True<\/span>\n<span class=\"k\">assert<\/span> <span class=\"mi\">1<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">1<\/span>\n<span class=\"k\">assert<\/span> <span class=\"p\">[<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"mi\">3<\/span><span class=\"p\">]<\/span>\n<span class=\"n\">a<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">1<\/span>\n<span class=\"n\">b<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">1<\/span>\n<span class=\"k\">assert<\/span> <span class=\"n\">a<\/span> <span class=\"o\">==<\/span> <span class=\"n\">b<\/span>\n\n<span class=\"c1\"># All throw AssertionError<\/span>\n<span class=\"k\">assert<\/span> <span class=\"kc\">False<\/span>\n<span class=\"k\">assert<\/span> <span class=\"mi\">1<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">2<\/span>\n<span class=\"k\">assert<\/span> <span class=\"p\">[]<\/span>\n<span class=\"n\">a<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">1<\/span>\n<span class=\"n\">b<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">2<\/span>\n<span class=\"k\">assert<\/span> <span class=\"n\">a<\/span> <span class=\"o\">==<\/span> <span class=\"n\">b<\/span>\n\n<span class=\"c1\"># You can include messages<\/span>\n<span class=\"k\">assert<\/span> <span class=\"kc\">False<\/span><span class=\"p\">,<\/span> <span class=\"s1\">&#39;This is forbidden&#39;<\/span>\n<span class=\"c1\"># Throws AssertionError: This is forbidden<\/span>\n<\/code><\/pre><\/div>\n\n<p>So how do you use this practically? Well, in a Django view, you can check all sorts of things that you believe are true. Check the assertions that you believe <em>maybe<\/em> aren't true, even though they <em>should<\/em> be.<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># views.py<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">my_view<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"n\">Thing<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span><span class=\"o\">.<\/span><span class=\"n\">exists<\/span><span class=\"p\">(),<\/span> <span class=\"s1\">&#39;there must be at least 1 thing in db&#39;<\/span>\n    <span class=\"n\">thing<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Things<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span><span class=\"o\">.<\/span><span class=\"n\">last<\/span><span class=\"p\">()<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"n\">thing<\/span><span class=\"p\">,<\/span> <span class=\"s1\">&#39;thing must exist&#39;<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"n\">thing<\/span><span class=\"o\">.<\/span><span class=\"n\">name<\/span><span class=\"p\">,<\/span> <span class=\"s1\">&#39;thing must have name&#39;<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"nb\">type<\/span><span class=\"p\">(<\/span><span class=\"n\">thing<\/span><span class=\"o\">.<\/span><span class=\"n\">name<\/span><span class=\"p\">)<\/span> <span class=\"ow\">is<\/span> <span class=\"nb\">str<\/span><span class=\"p\">,<\/span> <span class=\"s1\">&#39;thing name must be a str&#39;<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">HttpResponse<\/span><span class=\"p\">(<\/span><span class=\"sa\">f<\/span><span class=\"s2\">&quot;The thing is called <\/span><span class=\"si\">{<\/span><span class=\"n\">thing<\/span><span class=\"o\">.<\/span><span class=\"n\">name<\/span><span class=\"si\">}<\/span><span class=\"s2\">&quot;<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n<p>Deciding when to use print vs. assert vs. pdb comes with experience, so I recommend you give them all a try so that you can get a feel for them. These three methods are quick and simple to implement, wheras this final tip is the most useful, but also requires the most labour.<\/p>\n<h3>Reproduce the bug with tests<\/h3>\n<p>Some bugs can be quite tricky to reproduce. To trigger the line of code that causes the bug you might need to create a new user, log in as that user, verify their email, sign in, sign out, sign in again, buy their first product... etc. etc. etc. you get the idea.<\/p>\n<p>Even worse, you might have to do this series of steps dozens of times before you've fixed the bug. To avoid all of this hard work... you're going to have to do a little bit of hard work and write a test.<\/p>\n<p>The bad thing about tests is that they take some time to write. The good thing about tests is that you set up the data required to run the test once, and then you've automated the process forever. Tests become more valuable the more you run them, and you can run them <em>a lot<\/em>:<\/p>\n<ul>\n<li>You can quickly re-run them to reproduce the issue<\/li>\n<li>You can run them to check that the issue is solved<\/li>\n<li>You can run them in the future to make sure that the issue never comes back<\/li>\n<\/ul>\n<p>I'll give you a quick example. Say your issue is that when you call the view <code>my_view<\/code>, you get an error:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># views.py<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">my_view<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">thing<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Things<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span><span class=\"o\">.<\/span><span class=\"n\">last<\/span><span class=\"p\">()<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">HttpResponse<\/span><span class=\"p\">(<\/span><span class=\"sa\">f<\/span><span class=\"s2\">&quot;The thing is called <\/span><span class=\"si\">{<\/span><span class=\"n\">thing<\/span><span class=\"o\">.<\/span><span class=\"n\">name<\/span><span class=\"si\">}<\/span><span class=\"s2\">&quot;<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n<p>The error is<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>AttributeError: &#39;NoneType&#39; object has no attribute &#39;name&#39;\n<\/code><\/pre><\/div>\n\n<p>A quick test to run this view (using <a href=\"https:\/\/docs.pytest.org\/en\/latest\/\">pytest<\/a>) is:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># tests.py<\/span>\n\n<span class=\"nd\">@pytest<\/span><span class=\"o\">.<\/span><span class=\"n\">mark<\/span><span class=\"o\">.<\/span><span class=\"n\">django_db<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">test_my_view__with_thing<\/span><span class=\"p\">(<\/span><span class=\"n\">client<\/span><span class=\"p\">):<\/span>\n    <span class=\"sd\">&quot;&quot;&quot;<\/span>\n<span class=\"sd\">    Check that my_view returns thing name when there is a Thing<\/span>\n<span class=\"sd\">    &quot;&quot;&quot;<\/span>\n    <span class=\"n\">Thing<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span><span class=\"o\">.<\/span><span class=\"n\">create<\/span><span class=\"p\">(<\/span><span class=\"n\">name<\/span><span class=\"o\">=<\/span><span class=\"s2\">&quot;a thing&quot;<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">url<\/span> <span class=\"o\">=<\/span> <span class=\"n\">reverse<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;my-view&quot;<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">response<\/span> <span class=\"o\">=<\/span> <span class=\"n\">client<\/span><span class=\"o\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">(<\/span><span class=\"n\">url<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"n\">response<\/span><span class=\"o\">.<\/span><span class=\"n\">status_code<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">200<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"n\">response<\/span><span class=\"o\">.<\/span><span class=\"n\">data<\/span> <span class=\"o\">==<\/span> <span class=\"s2\">&quot;The thing is called a thing&quot;<\/span>\n\n\n<span class=\"nd\">@pytest<\/span><span class=\"o\">.<\/span><span class=\"n\">mark<\/span><span class=\"o\">.<\/span><span class=\"n\">django_db<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">test_my_view__with_no_thing<\/span><span class=\"p\">(<\/span><span class=\"n\">client<\/span><span class=\"p\">):<\/span>\n    <span class=\"sd\">&quot;&quot;&quot;<\/span>\n<span class=\"sd\">    Check that my_view returns no thing name when there is no Thing<\/span>\n<span class=\"sd\">    &quot;&quot;&quot;<\/span>\n    <span class=\"n\">url<\/span> <span class=\"o\">=<\/span> <span class=\"n\">reverse<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;my-view&quot;<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">response<\/span> <span class=\"o\">=<\/span> <span class=\"n\">client<\/span><span class=\"o\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">(<\/span><span class=\"n\">url<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"n\">response<\/span><span class=\"o\">.<\/span><span class=\"n\">status_code<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">200<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"n\">response<\/span><span class=\"o\">.<\/span><span class=\"n\">data<\/span> <span class=\"o\">==<\/span> <span class=\"s2\">&quot;The thing is called &quot;<\/span>\n<\/code><\/pre><\/div>\n\n<p>Note that even just writing these tests will show you where the code is broken, but this is just an example, so let's ignore that.<\/p>\n<p>When you run these tests, you'll notice that:<\/p>\n<ul>\n<li><code>test_my_view__with_thing<\/code> passes<\/li>\n<li><code>test_my_view__with_no_thing<\/code> fails, with an <code>AttributeError<\/code><\/li>\n<\/ul>\n<p>Now that we've nailed down the issue with a test, we can fix the bug, update the test and re-run it to make sure the bug is fixed. Now we've automated the process of reproducing the bug and checking that it's fixed.<\/p>\n<h3>Conclusion<\/h3>\n<p>So there you go, four tips for debugging Django:<\/p>\n<ul>\n<li>better print statements with <code>__dict__<\/code><\/li>\n<li>Python's pdb debugger<\/li>\n<li>assert statements<\/li>\n<li>reproducing the issue with tests<\/li>\n<\/ul>\n<p>Of all these four, I recommend you invest time into learning how to write tests. Effective testing has huge bang-for-buck, not just for debugging, but also for preventing bugs in the first place.<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"Quickly fix bugs in Django with Python's debugger","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/django-debug-pdb.html","rel":"alternate"}},"published":"2020-04-11T12:00:00+10:00","updated":"2020-04-11T12:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-04-11:\/django-debug-pdb.html","summary":"<p>There's a bug in your Django code. You've tried to track down the problem with \"print\" statements, but it's such a slow, tedious process:<\/p>\n<ul>\n<li>Add a \"print\" statement to your code<\/li>\n<li>Refresh the page in your browser to re-run your code<\/li>\n<li>Look at the <code>runserver<\/code> console output for the \"print \u2026<\/li><\/ul>","content":"<p>There's a bug in your Django code. You've tried to track down the problem with \"print\" statements, but it's such a slow, tedious process:<\/p>\n<ul>\n<li>Add a \"print\" statement to your code<\/li>\n<li>Refresh the page in your browser to re-run your code<\/li>\n<li>Look at the <code>runserver<\/code> console output for the \"print\" results<\/li>\n<\/ul>\n<p>Repeat this 100 times, maybe you find the issue. Is there a faster way to find and fix bugs in Django?<\/p>\n<h3>Python's built-in debugger<\/h3>\n<p>Python's standard library comes with a debugging tool and it is easily the most efficient tool for diving into your code and figuring out what's happening. Using the debugger is as simple as taking a Django view like this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># views.py<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">some_view<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">):<\/span>\n    <span class=\"sd\">&quot;&quot;&quot;Shows user some stuff&quot;&quot;&quot;<\/span>\n    <span class=\"n\">things<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Thing<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span><span class=\"o\">.<\/span><span class=\"n\">all<\/span><span class=\"p\">()<\/span>\n    <span class=\"n\">stuff<\/span> <span class=\"o\">=<\/span> <span class=\"n\">get_stuff<\/span><span class=\"p\">(<\/span><span class=\"n\">things<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">HttpResponse<\/span><span class=\"p\">(<\/span><span class=\"sa\">f<\/span><span class=\"s2\">&quot;The stuff is <\/span><span class=\"si\">{<\/span><span class=\"n\">stuff<\/span><span class=\"si\">}<\/span><span class=\"s2\">&quot;<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n<p>... and then whacking a single line of code into the view:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># views.py<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">my_view<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">):<\/span>\n    <span class=\"sd\">&quot;&quot;&quot;Shows user some stuff&quot;&quot;&quot;<\/span>\n    <span class=\"n\">things<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Thing<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span><span class=\"o\">.<\/span><span class=\"n\">all<\/span><span class=\"p\">()<\/span>\n\n    <span class=\"c1\"># Start debugging here<\/span>\n    <span class=\"kn\">import<\/span> <span class=\"nn\">pdb<\/span><span class=\"p\">;<\/span><span class=\"n\">pdb<\/span><span class=\"o\">.<\/span><span class=\"n\">set_trace<\/span><span class=\"p\">()<\/span>\n\n    <span class=\"n\">stuff<\/span> <span class=\"o\">=<\/span> <span class=\"n\">get_stuff<\/span><span class=\"p\">(<\/span><span class=\"n\">things<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">HttpResponse<\/span><span class=\"p\">(<\/span><span class=\"sa\">f<\/span><span class=\"s2\">&quot;The stuff is <\/span><span class=\"si\">{<\/span><span class=\"n\">stuff<\/span><span class=\"si\">}<\/span><span class=\"s2\">&quot;<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n<p>That's it, you're now using Python's debugger.<\/p>\n<h3>Yeah, but, what's it do?<\/h3>\n<p>Here's a short video I made showing you an example of using pdb in a Django view:<\/p>\n<div class=\"loom-embed\"><iframe src=\"https:\/\/www.loom.com\/embed\/7de384817fbc45f0918995646b199055\" frameborder=\"0\" webkitallowfullscreen mozallowfullscreen allowfullscreen style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%;\"><\/iframe><\/div>\n\n<h3>Quick reference<\/h3>\n<p>The <a href=\"https:\/\/docs.python.org\/3\/library\/pdb.html\">Python pdb docs<\/a> tell you all the commands, but for completeness, here are the commands I used:<\/p>\n<ul>\n<li><code>__dict__<\/code> - print Python object attributes as a dictionary<\/li>\n<li><code>type()<\/code> - print object type<\/li>\n<li><code>l \/ ll<\/code> - show the current line of code<\/li>\n<li><code>n<\/code> - execute next line<\/li>\n<li><code>s<\/code> - step inside function<\/li>\n<li><code>c<\/code> - exit debugger, continue running code<\/li>\n<li><code>q<\/code> - quit debugger, throw an exception<\/li>\n<\/ul>\n<p>Some extra commands worth trying, which I didn't show you:<\/p>\n<ul>\n<li><code>help<\/code> - print debugger help<\/li>\n<li><code>dir()<\/code> - print Python object functions available<\/li>\n<li><code>locals()<\/code> - print local variables<\/li>\n<li><code>globals()<\/code> - print global variables<\/li>\n<\/ul>\n<h3>Why the command line?<\/h3>\n<p>You might be wondering why I insist on using pdb from the command line rather than using some fancy integrated IDE like PyCharm or Visual Studio. Basically I think these tools take too long to set up. Using pdb requires no set up time with nothing to install. If you use an IDE-based debugger, then anytime you switch editors you'll need to set up your debugging tools. You don't want to waste time debugging your debugger. No thanks!<\/p>\n<h3>Bonus tip: run debugger on any exception<\/h3>\n<p>You can also set up pdb to start running anytime there is an exception:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>python -m pdb -c <span class=\"k\">continue<\/span> myscript.py\n<\/code><\/pre><\/div>\n\n<p>This doesn't work for Django, because of the way <code>runserver<\/code> handles exceptions, but you can use it for your other Python scripting.<\/p>\n<p>If you're testing Django with pytest you can force the testing tool to drop into the pdb debugger when it hits an error:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>pytest --pdb\n<\/code><\/pre><\/div>\n\n<h3>Next steps<\/h3>\n<p>Go out there and use pdb - it's one line of code! If you really want to step up your debugging, then I recommend learning how to write tests that reproduce your issue, and then use pdb in concert with your tests to find a fix, and make sure it stays fixed.<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"How to view Django logs with Papertrail","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/django-logging-papertrail.html","rel":"alternate"}},"published":"2020-04-10T12:00:00+10:00","updated":"2020-04-10T12:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-04-10:\/django-logging-papertrail.html","summary":"<p>You have a Django app running on a webserver and hopefully you're <a href=\"https:\/\/mattsegal.dev\/file-logging-django.html\">writing your logs to a file<\/a>. If anything goes wrong you can search back through the logs and figure out what happened.<\/p>\n<p>The problem is that to get to your logs, you have to log into your server \u2026<\/p>","content":"<p>You have a Django app running on a webserver and hopefully you're <a href=\"https:\/\/mattsegal.dev\/file-logging-django.html\">writing your logs to a file<\/a>. If anything goes wrong you can search back through the logs and figure out what happened.<\/p>\n<p>The problem is that to get to your logs, you have to log into your server, find the right file and search through the text on the command line. It's possible to do but it's kind of a pain. Isn't there an easier way to view your Django app's logs? Wouldn't it be nice to search through them on a website?<\/p>\n<p>This post will show you how to push your Django logs into <a href=\"https:\/\/www.papertrail.com\/\">Papertrail<\/a>. Papertrail is a free web-based log aggregator that is reasonably simple to set up. It stores ~6 days of searchable logs. It's best for small, simple projects where you don't want to do anything complicated.<\/p>\n<div class=\"loom-embed\"><iframe src=\"https:\/\/www.loom.com\/embed\/5ede7f70b62645ca82c1ffbf4c0e64eb\" frameborder=\"0\" webkitallowfullscreen mozallowfullscreen allowfullscreen style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%;\"><\/iframe><\/div>\n\n<h3>Create an account<\/h3>\n<p>You can start by going to the <a href=\"https:\/\/www.papertrail.com\/\">Papertrail website<\/a> and creating an account. Once that's done, you can visit <a href=\"https:\/\/papertrailapp.com\/systems\/setup?type=app&amp;platform=unix\">this page<\/a>, where you'll see a message like this:<\/p>\n<blockquote>\n<p>Your logs will go to logs2.papertrailapp.com:41234 and appear in Events.<\/p>\n<\/blockquote>\n<p>You need to note down two things from this page:<\/p>\n<ul>\n<li>The hostname: logs2.papertrailapp.com<\/li>\n<li>The port: 41234<\/li>\n<\/ul>\n<p>These two peices of information will determine where Papertrail stores your logs, and they're essentially secrets that should be kept out of public view. Keep the page open, because it'll be useful later.<\/p>\n<h3>Install Papertrail's remote_syslog2<\/h3>\n<p>Papertrail uses some tool they've built called <code>remote_syslog2<\/code> to ship logs from your server into their storage. Assuming you're running Ubuntu or Debian, you can download the .deb installation file for remote_syslog2 from GitHub. As of the writing of this post, <a href=\"https:\/\/github.com\/papertrail\/remote_syslog2\/releases\/download\/v0.20\/remote-syslog2_0.20_amd64.deb\">this is the latest release deb file<\/a>.<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># Download installation file to \/tmp\/<\/span>\n<span class=\"nv\">DEB_URL<\/span><span class=\"o\">=<\/span><span class=\"s2\">&quot; https:\/\/github.com\/papertrail\/...deb&quot;<\/span>\ncurl --location --silent <span class=\"nv\">$DEB_URL<\/span> -o \/tmp\/remote_syslog.deb\n\n<span class=\"c1\"># Install remote_syslog2 from the file<\/span>\nsudo dpkg -i \/tmp\/remote_syslog.deb\n<\/code><\/pre><\/div>\n\n<p>You can read more about remote_syslog <a href=\"https:\/\/help.papertrailapp.com\/kb\/configuration\/configuring-centralized-logging-from-text-log-files-in-unix\/\">here<\/a>.<\/p>\n<h3>Create logging config<\/h3>\n<p>You can configure what logs get sent to Papertrail using a config file. This uses the YAML format and should live at <code>\/etc\/log_files.yml<\/code><\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># \/etc\/log_files.yml<\/span><span class=\"w\"><\/span>\n<span class=\"nt\">files<\/span><span class=\"p\">:<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"p p-Indicator\">-<\/span><span class=\"w\"> <\/span><span class=\"l l-Scalar l-Scalar-Plain\">\/tmp\/papertrail-test.log<\/span><span class=\"w\"><\/span>\n<span class=\"nt\">destination<\/span><span class=\"p\">:<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"nt\">host<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"l l-Scalar l-Scalar-Plain\">logs2.papertrailapp.com<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"nt\">port<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"l l-Scalar l-Scalar-Plain\">41234<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"nt\">protocol<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"l l-Scalar l-Scalar-Plain\">tls<\/span><span class=\"w\"><\/span>\n<\/code><\/pre><\/div>\n\n<h3>Run Papertrail with a test log file<\/h3>\n<p>Start by testing out whether remote_syslog is setup correctly by running it in non-daemonized mode:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>remote_syslog -D --hostname myapp\n<\/code><\/pre><\/div>\n\n<p>Note that \"hostname\" can be whatever name you want. You should see some console output like this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>... Connecting to logs2.papertrailapp.com:41234 over tls\n... Cannot forward \/tmp\/papertrail-test.log, it may not exist\n<\/code><\/pre><\/div>\n\n<p>Make sure you have <a href=\"https:\/\/papertrailapp.com\/systems\/setup?type=app&amp;platform=unix\">this page<\/a> open in your web browser (or open it now). In another bash terminal, write some text to papertrail-test.log:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"nb\">echo<\/span> <span class=\"s2\">&quot;[<\/span><span class=\"k\">$(<\/span>date<span class=\"k\">)<\/span><span class=\"s2\">] Test logline&quot;<\/span> &gt;&gt; \/tmp\/papertrail-test.log\n<\/code><\/pre><\/div>\n\n<p>Now you should see, in your remote_syslog terminal, a new message:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>... Forwarding file: \/tmp\/papertrail-test.log\n<\/code><\/pre><\/div>\n\n<p>When you look at the page you have open, you should see something like:<\/p>\n<blockquote>\n<p>Logs received from myapp<\/p>\n<\/blockquote>\n<p>If you head to your <a href=\"https:\/\/papertrailapp.com\/dashboard\">dashboard<\/a> you should now see a new system added called \"myapp\". You should be also able to see your test log messages in the <a href=\"https:\/\/my.papertrailapp.com\/systems\/myapp\/events\">search panel for myapp<\/a>.<\/p>\n<h3>Run Papertrail with real log files<\/h3>\n<p>Now that you're happy that Papertrail is able to upload log messages, you can set it up to ship your log files. In this example, I'm going to upload data from the Django and gunicorn log files I created in <a href=\"https:\/\/mattsegal.dev\/file-logging-django.html\">this post<\/a>:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># \/etc\/log_files.yml<\/span><span class=\"w\"><\/span>\n<span class=\"nt\">files<\/span><span class=\"p\">:<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"p p-Indicator\">-<\/span><span class=\"w\"> <\/span><span class=\"l l-Scalar l-Scalar-Plain\">\/var\/log\/django.log<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"p p-Indicator\">-<\/span><span class=\"w\"> <\/span><span class=\"l l-Scalar l-Scalar-Plain\">\/var\/log\/gunicorn\/access.log<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"p p-Indicator\">-<\/span><span class=\"w\"> <\/span><span class=\"l l-Scalar l-Scalar-Plain\">\/var\/log\/gunicorn\/error.log<\/span><span class=\"w\"><\/span>\n<span class=\"nt\">destination<\/span><span class=\"p\">:<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"nt\">host<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"l l-Scalar l-Scalar-Plain\">logs2.papertrailapp.com<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"nt\">port<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"l l-Scalar l-Scalar-Plain\">41234<\/span><span class=\"w\"><\/span>\n<span class=\"w\">  <\/span><span class=\"nt\">protocol<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"l l-Scalar l-Scalar-Plain\">tls<\/span><span class=\"w\"><\/span>\n<\/code><\/pre><\/div>\n\n<p>When you are not testing with remote_syslog, you want to run it in daemonized mode:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>sudo remote_syslog --hostname myapp\n<\/code><\/pre><\/div>\n\n<p>You can check that it's still running by looking up its process:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>ps aux <span class=\"p\">|<\/span> grep remote_syslog\n<\/code><\/pre><\/div>\n\n<p>If you need to stop it:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>pkill remote_syslog\n<\/code><\/pre><\/div>\n\n<p>That's it! Now you have remote_syslog running on your server, shipping log data off to Papertrail.<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"How to save Django logs in production","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/file-logging-django.html","rel":"alternate"}},"published":"2020-04-10T12:00:00+10:00","updated":"2020-04-10T12:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-04-10:\/file-logging-django.html","summary":"<p>You've deployed Django to a webserver and something has broken. There's an error <em>somewhere<\/em>. What happened? When you're debugging Django on your local computer, you can just throw a print statement into your code and check the output in the runserver logs. What about in production? Where do the logs \u2026<\/p>","content":"<p>You've deployed Django to a webserver and something has broken. There's an error <em>somewhere<\/em>. What happened? When you're debugging Django on your local computer, you can just throw a print statement into your code and check the output in the runserver logs. What about in production? Where do the logs go there? How can I set up Django so it's easy to see what is happening?<\/p>\n<h2>Write your logs to a file<\/h2>\n<p>You need to get your deployed Django app to write its logs to a file, so that you can look at them later. You can do this by configuring Django's settings. You will also need to use Python's logging library, rather than print statements. Why use logging over print? The logging library generally makes it easier to manage logs in production. Specifically, it makes it easier to:<\/p>\n<ul>\n<li>write logs to a file<\/li>\n<li>track extra data like the current time and function<\/li>\n<li>filter your logs<\/li>\n<\/ul>\n<p>You might be thinking that using \"print\" works fine when you're using Django's dev web server. It's true! Using \"print\" works fine locally, but when you're in production with DEBUG=False, you won't be able to see your print statements anymore in Django's log output. Log messages will still show up when you're working locally so there's nothing to lose by ditching print for logging.<\/p>\n<h2>How to use logging in Django<\/h2>\n<p>Before you set Django up to write logs to a file, you need to use Python's logging framework to write any log messages that you want to record. It's pretty easy, you just need to set it up in each module that needs it. For example, in one of your views:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># views.py<\/span>\n<span class=\"c1\"># Import logging from Python&#39;s standard library<\/span>\n<span class=\"kn\">import<\/span> <span class=\"nn\">logging<\/span>\n\n<span class=\"c1\"># Create a logger for this file<\/span>\n<span class=\"n\">logger<\/span> <span class=\"o\">=<\/span> <span class=\"n\">logging<\/span><span class=\"o\">.<\/span><span class=\"n\">getLogger<\/span><span class=\"p\">(<\/span><span class=\"vm\">__file__<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">some_view<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">):<\/span>\n    <span class=\"sd\">&quot;&quot;&quot;<\/span>\n<span class=\"sd\">    Example view showing all the ways you can log messages.<\/span>\n<span class=\"sd\">    &quot;&quot;&quot;<\/span>\n    <span class=\"n\">logger<\/span><span class=\"o\">.<\/span><span class=\"n\">debug<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;This logs a debug message.&quot;<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">logger<\/span><span class=\"o\">.<\/span><span class=\"n\">info<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;This logs an info message.&quot;<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">logger<\/span><span class=\"o\">.<\/span><span class=\"n\">warn<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;This logs a warning message.&quot;<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">logger<\/span><span class=\"o\">.<\/span><span class=\"n\">error<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;This logs an error message.&quot;<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">try<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">raise<\/span> <span class=\"ne\">Exception<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;This is a handled exception&quot;<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">except<\/span> <span class=\"ne\">Exception<\/span><span class=\"p\">:<\/span>\n        <span class=\"n\">logger<\/span><span class=\"o\">.<\/span><span class=\"n\">exception<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;This logs an exception.&quot;<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"k\">raise<\/span> <span class=\"ne\">Exception<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;This is an unhandled exception&quot;<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">HttpResponse<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;this worked&quot;<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n<p>Most of the time I just use:<\/p>\n<ul>\n<li>logger.info for events I want to track, like a purchase being made<\/li>\n<li>logger.error for logical errors, like things that should never happen according to business rules<\/li>\n<li>logger.exception for when I catch an exception<\/li>\n<\/ul>\n<p>Once you've configured your logging in your settings (shown further below), you'll see messages like this appear in your log file (thanks to the info, warn and error methods):<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>2020-04-10 03:35:05 [INFO    ] (views.some_view) This logs an info message.\n2020-04-10 03:35:05 [WARNING ] (views.some_view) This logs a warn message.\n2020-04-10 03:35:05 [ERROR   ] (views.some_view) This logs an error message.\n<\/code><\/pre><\/div>\n\n<p>And you'll see your message plus a stack trace when you log using the exeption method:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>2020-04-10 03:35:05 [ERROR   ] (views.some_view) This logs an exception.\nTraceback (most recent call last):\n  File &quot;...\/myproj\/views.py&quot;, line 14, in log_view\n    raise Exception(&quot;This is a handled exception&quot;)\nException: This is a handled exception\n<\/code><\/pre><\/div>\n\n<p>And you'll still get an error log and stack track for your unhandled exceptions:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>2020-04-10 03:35:05 [ERROR   ] (log.log_response) Internal Server Error:\nTraceback (most recent call last):\n  File &quot;...\/exception.py&quot;, line 34, in inner\n    response = get_response(request)\n  File &quot;...\/base.py&quot;, line 115, in _get_response\n    response = self.process_exception_by_middleware(e, request)\n  File &quot;...\/base.py&quot;, line 113, in _get_response\n    response = wrapped_callback(request, *callback_args, **callback_kwargs)\n  File &quot;...\/myproj\/views.py&quot;, line 18, in log_view\n    raise Exception(&quot;This is an unhandled exception&quot;)\nException: This is an unhandled exception\n<\/code><\/pre><\/div>\n\n<p>Importantly, you won't see any results from print statements, which is why you can't use them for production logging.<\/p>\n<div class=\"ui divider\" style=\"margin: 1.5em 0;\"><\/div>\n<form action=\"https:\/\/dev.us19.list-manage.com\/subscribe\/post?u=e7a1ec466f7bb1732dbd23fc7&amp;id=ec345473bd\" method=\"post\" name=\"mc-embedded-subscribe-form\" target=\"_blank\" style=\"text-align: center; padding-bottom: 1em;\" novalidate>\n  <h3 class=\"subscribe-cta\">Get alerted when I publish new blog posts<\/h3>\n  <div class=\"ui fluid action input subscribe\">\n    <input\n      type=\"email\"\n      value=\"\"\n      name=\"EMAIL\"\n      placeholder=\"Enter your email address\"\n    \/>\n    <button class=\"ui primary button\" type=\"submit\" name=\"subscribe\">\n      Subscribe\n    <\/button>\n  <\/div>\n  <div style=\"position: absolute; left: -5000px;\" aria-hidden=\"true\">\n    <input\n      type=\"text\"\n      name=\"b_e7a1ec466f7bb1732dbd23fc7_ec345473bd\"\n      tabindex=\"-1\"\n      value=\"\"\n    \/>\n  <\/div>\n<\/form>\n<div class=\"ui divider\" style=\"margin: 1.5em 0;\"><\/div>\n\n<h2>How to set up file logging<\/h2>\n<p>Now that you're sold on logging and you know how to use it in your code, you can set it up in your Django settings.<\/p>\n<p>I like to do this by splitting my settings module up into two files - one for dev and one for production. Usually your Django project's main app will have your settings set up something like this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>myapp\n\u251c\u2500\u2500 settings.py\n\u251c\u2500\u2500 urls.py\n\u2514\u2500\u2500 wsgi.py\n<\/code><\/pre><\/div>\n\n<p>I recommend turning settings into a folder, and moving the original settings.py file into the folder's __init__.py file:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>myapp\n\u251c\u2500\u2500 settings\n|   \u251c\u2500\u2500 __init__.py\n|   \u2514\u2500\u2500 prod.py\n\u251c\u2500\u2500 urls.py\n\u2514\u2500\u2500 wsgi.py\n<\/code><\/pre><\/div>\n\n<p>So that __init__.py has all your original settings<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># __init__.py<\/span>\n<span class=\"c1\"># Base settings for myapp<\/span>\n<span class=\"kn\">import<\/span> <span class=\"nn\">os<\/span>\n\n<span class=\"n\">BASE_DIR<\/span> <span class=\"o\">=<\/span> <span class=\"n\">os<\/span><span class=\"o\">.<\/span><span class=\"n\">path<\/span><span class=\"o\">.<\/span><span class=\"n\">dirname<\/span><span class=\"p\">(<\/span><span class=\"n\">os<\/span><span class=\"o\">.<\/span><span class=\"n\">path<\/span><span class=\"o\">.<\/span><span class=\"n\">dirname<\/span><span class=\"p\">(<\/span><span class=\"n\">os<\/span><span class=\"o\">.<\/span><span class=\"n\">path<\/span><span class=\"o\">.<\/span><span class=\"n\">abspath<\/span><span class=\"p\">(<\/span><span class=\"vm\">__file__<\/span><span class=\"p\">)))<\/span>\n<span class=\"n\">SECRET_KEY<\/span> <span class=\"o\">=<\/span> <span class=\"s2\">&quot;xxx&quot;<\/span>\n<span class=\"n\">DEBUG<\/span> <span class=\"o\">=<\/span> <span class=\"kc\">True<\/span>\n<span class=\"n\">ALLOWED_HOSTS<\/span> <span class=\"o\">=<\/span> <span class=\"p\">[]<\/span>\n<span class=\"c1\"># ... all the regular Django settings ...<\/span>\n<\/code><\/pre><\/div>\n\n<p>and prod.py has your production-only settings:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># prod.py<\/span>\n<span class=\"c1\"># Production settings for myapp<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">.<\/span> <span class=\"kn\">import<\/span> <span class=\"o\">*<\/span>  <span class=\"c1\"># Import base settings from settings\/__init__.py<\/span>\n\n<span class=\"n\">ALLOWED_HOSTS<\/span> <span class=\"o\">=<\/span> <span class=\"p\">[<\/span><span class=\"s2\">&quot;www.myapp.com&quot;<\/span><span class=\"p\">]<\/span>\n<span class=\"n\">DEBUG<\/span> <span class=\"o\">=<\/span> <span class=\"kc\">False<\/span>\n<span class=\"c1\"># ... whatever else you need ...<\/span>\n<\/code><\/pre><\/div>\n\n<p>In this prod.py settings file, I recommend adding the following logging config:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"n\">LOGGING<\/span> <span class=\"o\">=<\/span> <span class=\"p\">{<\/span>\n    <span class=\"s2\">&quot;version&quot;<\/span><span class=\"p\">:<\/span> <span class=\"mi\">1<\/span><span class=\"p\">,<\/span>\n    <span class=\"s2\">&quot;disable_existing_loggers&quot;<\/span><span class=\"p\">:<\/span> <span class=\"kc\">False<\/span><span class=\"p\">,<\/span>\n    <span class=\"s2\">&quot;root&quot;<\/span><span class=\"p\">:<\/span> <span class=\"p\">{<\/span><span class=\"s2\">&quot;level&quot;<\/span><span class=\"p\">:<\/span> <span class=\"s2\">&quot;INFO&quot;<\/span><span class=\"p\">,<\/span> <span class=\"s2\">&quot;handlers&quot;<\/span><span class=\"p\">:<\/span> <span class=\"p\">[<\/span><span class=\"s2\">&quot;file&quot;<\/span><span class=\"p\">]},<\/span>\n    <span class=\"s2\">&quot;handlers&quot;<\/span><span class=\"p\">:<\/span> <span class=\"p\">{<\/span>\n        <span class=\"s2\">&quot;file&quot;<\/span><span class=\"p\">:<\/span> <span class=\"p\">{<\/span>\n            <span class=\"s2\">&quot;level&quot;<\/span><span class=\"p\">:<\/span> <span class=\"s2\">&quot;INFO&quot;<\/span><span class=\"p\">,<\/span>\n            <span class=\"s2\">&quot;class&quot;<\/span><span class=\"p\">:<\/span> <span class=\"s2\">&quot;logging.FileHandler&quot;<\/span><span class=\"p\">,<\/span>\n            <span class=\"s2\">&quot;filename&quot;<\/span><span class=\"p\">:<\/span> <span class=\"s2\">&quot;\/var\/log\/django.log&quot;<\/span><span class=\"p\">,<\/span>\n            <span class=\"s2\">&quot;formatter&quot;<\/span><span class=\"p\">:<\/span> <span class=\"s2\">&quot;app&quot;<\/span><span class=\"p\">,<\/span>\n        <span class=\"p\">},<\/span>\n    <span class=\"p\">},<\/span>\n    <span class=\"s2\">&quot;loggers&quot;<\/span><span class=\"p\">:<\/span> <span class=\"p\">{<\/span>\n        <span class=\"s2\">&quot;django&quot;<\/span><span class=\"p\">:<\/span> <span class=\"p\">{<\/span>\n            <span class=\"s2\">&quot;handlers&quot;<\/span><span class=\"p\">:<\/span> <span class=\"p\">[<\/span><span class=\"s2\">&quot;file&quot;<\/span><span class=\"p\">],<\/span>\n            <span class=\"s2\">&quot;level&quot;<\/span><span class=\"p\">:<\/span> <span class=\"s2\">&quot;INFO&quot;<\/span><span class=\"p\">,<\/span>\n            <span class=\"s2\">&quot;propagate&quot;<\/span><span class=\"p\">:<\/span> <span class=\"kc\">True<\/span>\n        <span class=\"p\">},<\/span>\n    <span class=\"p\">},<\/span>\n    <span class=\"s2\">&quot;formatters&quot;<\/span><span class=\"p\">:<\/span> <span class=\"p\">{<\/span>\n        <span class=\"s2\">&quot;app&quot;<\/span><span class=\"p\">:<\/span> <span class=\"p\">{<\/span>\n            <span class=\"s2\">&quot;format&quot;<\/span><span class=\"p\">:<\/span> <span class=\"p\">(<\/span>\n                <span class=\"sa\">u<\/span><span class=\"s2\">&quot;<\/span><span class=\"si\">%(asctime)s<\/span><span class=\"s2\"> [<\/span><span class=\"si\">%(levelname)-8s<\/span><span class=\"s2\">] &quot;<\/span>\n                <span class=\"s2\">&quot;(<\/span><span class=\"si\">%(module)s<\/span><span class=\"s2\">.<\/span><span class=\"si\">%(funcName)s<\/span><span class=\"s2\">) <\/span><span class=\"si\">%(message)s<\/span><span class=\"s2\">&quot;<\/span>\n            <span class=\"p\">),<\/span>\n            <span class=\"s2\">&quot;datefmt&quot;<\/span><span class=\"p\">:<\/span> <span class=\"s2\">&quot;%Y-%m-<\/span><span class=\"si\">%d<\/span><span class=\"s2\"> %H:%M:%S&quot;<\/span><span class=\"p\">,<\/span>\n        <span class=\"p\">},<\/span>\n    <span class=\"p\">},<\/span>\n<span class=\"p\">}<\/span>\n<\/code><\/pre><\/div>\n\n<p>This is, admittedly, a horrific clusterfuck of configuration. It burns my eyes and I imagine it makes you want to slam your laptop shut and run away screaming. If you want to know how it all works, I recommend watching <a href=\"https:\/\/www.youtube.com\/watch?v=DxZ5WEo4hvU\">this presentation<\/a>. If not, feel free to blindly copy now and figure it out later.<\/p>\n<p>The relevant area for you is in <code>LOGGING[\"handlers\"][\"file\"]<\/code>. This dict defines the bit that acutally writes our logs to the file. The important key is \"filename\", which defines the filepath where your logs will be written. You might want to change this depending on your preferences.<\/p>\n<h2>Use prod settings in production<\/h2>\n<p>The last little trick you need is to tell Django to use your prod settings in production. You can do this a few ways, I like to do it by setting the DJANGO_SETTINGS_MODULE environment variable.<\/p>\n<p>When I launch gunicorn, I do this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># Set Django settings to use prod.py<\/span>\n<span class=\"nb\">export<\/span> <span class=\"nv\">DJANGO_SETTINGS_MODULE<\/span><span class=\"o\">=<\/span>myproj.settings.prod\n\n<span class=\"c1\"># Launch gunicorn as-per-normal<\/span>\ngunicorn myproj.wsgi:application\n<\/code><\/pre><\/div>\n\n<h2>Bonus round: gunicorn logs<\/h2>\n<p>If you're using gunicorn as your WSGI app server in production, you might also want to track your gunicorn logs. This will give you information about incoming web requests, and the app starting and stopping, which can be useful when debugging. To do this, you just need to set some command-line flags:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># Set Django settings to use prod.py<\/span>\n<span class=\"nb\">export<\/span> <span class=\"nv\">DJANGO_SETTINGS_MODULE<\/span><span class=\"o\">=<\/span>myproj.settings.prod\n\n<span class=\"c1\"># Create logging folder for gunicorn.<\/span>\nmkdir -p \/var\/log\/gunicorn\n\n<span class=\"c1\"># Launch gunicorn with access and error logging.<\/span>\ngunicorn myproj.wsgi:application <span class=\"se\">\\<\/span>\n    --error-logfile \/var\/log\/gunicorn\/error.log <span class=\"se\">\\<\/span>\n    --access-logfile \/var\/log\/gunicorn\/access.log\n<\/code><\/pre><\/div>\n\n<p>Gunicorn's access logs look something like this, telling you about incoming web requests:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>127.0.0.1 - - [10\/Apr\/2020:02:46:09 +0000] &quot;GET \/logs\/ HTTP\/1.1&quot; 400 143 ...\n127.0.0.1 - - [10\/Apr\/2020:02:46:43 +0000] &quot;GET \/logs\/ HTTP\/1.1&quot; 500 145 ...\n<\/code><\/pre><\/div>\n\n<p>And the error logs are mostly information about the app booting up and stopping:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>[2020-04-10 12:45:57 +1000] [14814] [INFO] Starting gunicorn 20.0.4\n[2020-04-10 12:45:57 +1000] [14814] [INFO] Listening at: http:\/\/127.0.0.1:8000\n[2020-04-10 12:45:57 +1000] [14814] [INFO] Using worker: sync\n[2020-04-10 12:45:57 +1000] [14817] [INFO] Booting worker with pid: 14817\n[2020-04-10 12:46:38 +1000] [14814] [INFO] Handling signal: int\n[2020-04-10 02:46:38 +0000] [14817] [INFO] Worker exiting (pid: 14817)\n[2020-04-10 12:46:38 +1000] [14814] [INFO] Shutting down: Master\n<\/code><\/pre><\/div>\n\n<p>Both of these can be pretty useful when debugging issues in production.<\/p>\n<h2>Next steps<\/h2>\n<p>Once you've conifgured all of this, you'll be able to log into your webserver and see all your info events, error messages, access logs and gunicorn events. Finding and fixing an error in prod will be much easier with these logs.<\/p>\n<p>Wouldn't it be nice if you didn't have to log into production to see these messages though? Even better, wouldn't it be great to search through your logs? That's when log aggregation tools like Papertrail or SumoLogic come in handy. I've written a guide on how to set up Papertrail <a href=\"https:\/\/mattsegal.dev\/django-logging-papertrail.html\">here<\/a>.<\/p>\n<p>In addition, if you're running a professional operation, wouldn't it be good to get alerts when you have errors? That's when you need to <a href=\"https:\/\/mattsegal.dev\/sentry-for-django-error-monitoring.html\">set up error reporting<\/a> as well as logging.<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"How to customise a class based view in Django","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/customise-class-based-view-django.html","rel":"alternate"}},"published":"2020-04-09T12:00:00+10:00","updated":"2020-04-09T12:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-04-09:\/customise-class-based-view-django.html","summary":"<p>You've spend a little bit of time working on your Django app and you want to dip your toes into class-based views. The basic examples are simple enough, but once you want to do something more complicated, something more custom, you get stuck. How do you customise a class-based view \u2026<\/p>","content":"<p>You've spend a little bit of time working on your Django app and you want to dip your toes into class-based views. The basic examples are simple enough, but once you want to do something more complicated, something more custom, you get stuck. How do you customise a class-based view?<\/p>\n<p>You've written some function-based views before, and they seem pretty straightforward, it's just a function! If you want to change how it works, you just change the code inside the function. Simple - no magic, no mystery, it's just code. Customising class based views seems much less user-friendly.<\/p>\n<p>In this post I'll take you through a worked example, showing you how to customise class-based views.<\/p>\n<h3>Example problem<\/h3>\n<p>Let's start with an example problem. Say we've got a model called Article, used for publishing news online:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># models.py<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Article<\/span><span class=\"p\">(<\/span><span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">Model<\/span><span class=\"p\">):<\/span>\n\n    <span class=\"n\">created_at<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">DateTimeField<\/span><span class=\"p\">(<\/span><span class=\"n\">default<\/span><span class=\"o\">=<\/span><span class=\"n\">timezone<\/span><span class=\"o\">.<\/span><span class=\"n\">now<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">published_at<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">DateTimeField<\/span><span class=\"p\">(<\/span><span class=\"n\">blank<\/span><span class=\"o\">=<\/span><span class=\"kc\">True<\/span><span class=\"p\">,<\/span> <span class=\"n\">null<\/span><span class=\"o\">=<\/span><span class=\"kc\">True<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">title<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">CharField<\/span><span class=\"p\">(<\/span><span class=\"n\">max_length<\/span><span class=\"o\">=<\/span><span class=\"mi\">512<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">body_html<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">TextField<\/span><span class=\"p\">()<\/span>\n<\/code><\/pre><\/div>\n\n<p>We have a function-based view that lists all the articles:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># views.py<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">article_list_view<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">articles<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Article<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span><span class=\"o\">.<\/span><span class=\"n\">all<\/span><span class=\"p\">()<\/span>\n    <span class=\"n\">context<\/span> <span class=\"o\">=<\/span> <span class=\"p\">{<\/span><span class=\"s1\">&#39;articles&#39;<\/span><span class=\"p\">:<\/span> <span class=\"n\">articles<\/span><span class=\"p\">}<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">render<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">,<\/span> <span class=\"s1\">&#39;news\/article_list.html&#39;<\/span><span class=\"p\">,<\/span> <span class=\"n\">context<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n<p>As I mentioned earlier, this function-based code is pretty easy to customise - you just change the code! Let's say we only want to list all the <em>published<\/em> articles and list them from newest to oldest:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># views.py<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">article_list_view<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">articles<\/span> <span class=\"o\">=<\/span> <span class=\"p\">(<\/span>\n        <span class=\"n\">Article<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span>\n        <span class=\"o\">.<\/span><span class=\"n\">filter<\/span><span class=\"p\">(<\/span><span class=\"n\">published_at__isnull<\/span><span class=\"o\">=<\/span><span class=\"kc\">False<\/span><span class=\"p\">)<\/span>\n        <span class=\"o\">.<\/span><span class=\"n\">order_by<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;-published_at&#39;<\/span><span class=\"p\">)<\/span>\n    <span class=\"p\">)<\/span>\n    <span class=\"n\">context<\/span> <span class=\"o\">=<\/span> <span class=\"p\">{<\/span><span class=\"s1\">&#39;object_list&#39;<\/span><span class=\"p\">:<\/span> <span class=\"n\">articles<\/span><span class=\"p\">}<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">render<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">,<\/span> <span class=\"s2\">&quot;news\/article_list.html&quot;<\/span><span class=\"p\">,<\/span> <span class=\"n\">context<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n<p>Now let's try doing the same thing with a class-based view. Listing all Articles is <em>super<\/em> simple. It's like 3 lines of code:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># views<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">ArticleListView<\/span><span class=\"p\">(<\/span><span class=\"n\">ListView<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">model<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Article<\/span>\n    <span class=\"n\">template_name<\/span> <span class=\"o\">=<\/span> <span class=\"s2\">&quot;news\/article_list.html&quot;<\/span>\n<\/code><\/pre><\/div>\n\n<p>Cool, cool, and now we need to do the next bit: list all the <em>published<\/em> articles and list them from newest to oldest. How the fuck do we do that? Where do you even start? Are you stressed? I'm stressed.<\/p>\n<h3>The fix<\/h3>\n<p>The fix is to read some documentation. Not the <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/ref\/class-based-views\/\">Django docs<\/a>, which are great for a lot of topics. No, you are going to need to refer to <a href=\"https:\/\/ccbv.co.uk\/\">Classy Class-Based Views<\/a> to keep your sanity. Let's take a peek at the documentation for <a href=\"https:\/\/ccbv.co.uk\/projects\/Django\/3.0\/django.views.generic.list\/ListView\/\">ListView<\/a>.<\/p>\n<p>I'm going to cut to video to show you the rest of the fix.<\/p>\n<div class=\"loom-embed\"><iframe src=\"https:\/\/www.loom.com\/embed\/914ef155a98f49faba6c3c8af3d686a4\" frameborder=\"0\" webkitallowfullscreen mozallowfullscreen allowfullscreen style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%;\"><\/iframe><\/div>\n\n<p>You can use the techniques of overriding on any of the class-based view methods, depending on what you need to do.<\/p>\n<p>A common method to override is get_context_data:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"k\">class<\/span> <span class=\"nc\">ArticleListView<\/span><span class=\"p\">(<\/span><span class=\"n\">ListView<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">model<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Article<\/span>\n    <span class=\"n\">template_name<\/span> <span class=\"o\">=<\/span> <span class=\"s2\">&quot;news\/article_list.html&quot;<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">get_context_data<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"o\">**<\/span><span class=\"n\">kwargs<\/span><span class=\"p\">):<\/span>\n        <span class=\"n\">context<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">super<\/span><span class=\"p\">()<\/span><span class=\"o\">.<\/span><span class=\"n\">get_context_data<\/span><span class=\"p\">(<\/span><span class=\"o\">**<\/span><span class=\"n\">kwargs<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">return<\/span> <span class=\"p\">{<\/span>\n            <span class=\"o\">**<\/span><span class=\"n\">context<\/span><span class=\"p\">,<\/span>\n            <span class=\"s1\">&#39;now&#39;<\/span><span class=\"p\">:<\/span> <span class=\"n\">timezone<\/span><span class=\"o\">.<\/span><span class=\"n\">now<\/span><span class=\"p\">()<\/span>\n        <span class=\"p\">}<\/span>\n<\/code><\/pre><\/div>\n\n<p>In summary, when you're stuck on a class-based view:<\/p>\n<ul>\n<li>Go to <a href=\"https:\/\/ccbv.co.uk\/\">Classy Class-Based Views<\/a><\/li>\n<li>Take a peek at the attributes of the class<\/li>\n<li>Scan over the methods of the class<\/li>\n<li>Dig into the methods to figure out what you need to change<\/li>\n<li>Set any attributes that are necessary<\/li>\n<li>Override any methods that you need to change<\/li>\n<\/ul>","category":{"@attributes":{"term":"Django"}}},{"title":"Sentry is great for tracking Django errors","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/sentry-for-django-error-monitoring.html","rel":"alternate"}},"published":"2020-04-08T12:00:00+10:00","updated":"2020-04-08T12:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-04-08:\/sentry-for-django-error-monitoring.html","summary":"<p>You've deployed a Django app to a webserver and now it's not working. Your app is throwing 500 Internal Server Errors - what's wrong? Why is this happening? It worked on my laptop!?<\/p>\n<p>Even worse is when a <em>customer<\/em> experienced an error 12 hours ago and <em>you<\/em> need to figure out \u2026<\/p>","content":"<p>You've deployed a Django app to a webserver and now it's not working. Your app is throwing 500 Internal Server Errors - what's wrong? Why is this happening? It worked on my laptop!?<\/p>\n<p>Even worse is when a <em>customer<\/em> experienced an error 12 hours ago and <em>you<\/em> need to figure out what went wrong.<\/p>\n<h3>Error reporting<\/h3>\n<p>You need something to alert you when errors happen in production, otherwise you're flying blind. How can you fix a bug if you don't know what happened? Error reporting is important if you're a new developer, because you're going to write a lot of bugs, or if you're experienced, since other people are likely relying on your code to work.<\/p>\n<p>Django allows you to set up <a href=\"https:\/\/docs.djangoproject.com\/en\/3.0\/howto\/error-reporting\/\">email reports<\/a>, which requires some fiddling with mail servers, but it's a totally OK way to track errors.<\/p>\n<p>My favourite way to monitor errors is using <a href=\"https:\/\/sentry.io\/welcome\/\">Sentry<\/a>. It's a SaaS product that's been used at every Django job I've worked at and I use it for my personal projects. Here's why I like it so much.<\/p>\n<h3>Easy to set up<\/h3>\n<p>Sentry used to be a little harder to install, but now there are only 3 things you need to do in order to get started.<\/p>\n<p>Install the Python package<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>pip install sentry-sdk\n<\/code><\/pre><\/div>\n\n<p>Set an environment variable<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"nb\">export<\/span> <span class=\"nv\">SENTRY_DSN<\/span><span class=\"o\">=<\/span><span class=\"s2\">&quot;https:\/\/xxx@sentry.io\/yyy&quot;<\/span>\n<\/code><\/pre><\/div>\n\n<p>And run a line of Python in your <em>production<\/em> settings.py<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"kn\">import<\/span> <span class=\"nn\">sentry_sdk<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">sentry_sdk.integrations.django<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">DjangoIntegration<\/span>\n\n<span class=\"n\">sentry_sdk<\/span><span class=\"o\">.<\/span><span class=\"n\">init<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">dsn<\/span><span class=\"o\">=<\/span><span class=\"n\">os<\/span><span class=\"o\">.<\/span><span class=\"n\">environ<\/span><span class=\"o\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;SENTRY_DSN&quot;<\/span><span class=\"p\">),<\/span>\n    <span class=\"n\">integrations<\/span><span class=\"o\">=<\/span><span class=\"p\">[<\/span><span class=\"n\">DjangoIntegration<\/span><span class=\"p\">()],<\/span>\n    <span class=\"n\">environment<\/span><span class=\"o\">=<\/span><span class=\"s2\">&quot;prod&quot;<\/span>\n<span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n<h3>Rich reporting<\/h3>\n<p>You want as much information as possible on the incident:<\/p>\n<ul>\n<li>when did it happen?<\/li>\n<li>how many times did it happen?<\/li>\n<li>what URL was requested?<\/li>\n<li>what cookies were set?<\/li>\n<li>was it for a particular user?<\/li>\n<li>was it in a particular browser?<\/li>\n<li>what line of code triggered the error?<\/li>\n<li>what's the stack trace?<\/li>\n<\/ul>\n<p>Sentry captures an incredible amount of info when it logs an error, sometimes including the values of variables in scope and the database queries that were run.<\/p>\n<h3>Free for multiple apps<\/h3>\n<p>I host all my personal projects in Sentry for free. I think as long as you stick to one user you don't have to pay for it.<\/p>\n<h3>Handles frontend errors<\/h3>\n<p>You can use Sentry in your frontend JavaScript as well, which gives you a much more complete picture of what went wrong.<\/p>\n<h3>Slack integration<\/h3>\n<p>Some people might not like this, but if you pefer Slack to email, you can set up Sentry to post to a Slack channel when an error crops up.<\/p>\n<h3>Deployment tracking<\/h3>\n<p>If you're willing to do a little more legwork, you can configure Sentry to track your deployments. You give it a Git commit hash and it is able to correlate errors with particular deployments, making it easier to track down the offending code. This is particularly useful if you\/your team are shippng multiple deployments per day.<\/p>\n<h3>Wrapping up<\/h3>\n<p>As you might have noticed, I'm pretty happy with Sentry, even after a few years of using it. There are a few little issues with it, like the overly-complex settings panel in the web UI, but overall it offers a low-friction user experience. Hopefully you'll get some use out of it.<\/p>\n<p>Now, just because you have error monitoring set up, that doesn't mean you've done everything you need to in order to monitor your production environment. Application logging is essential as well! If you haven't already set up your Django app to write logs in production <a href=\"https:\/\/mattsegal.dev\/file-logging-django.html\">you can find out how here<\/a>.<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"3 ways to deploy a Django backend with a React frontend","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/django-spa-infrastructure.html","rel":"alternate"}},"published":"2020-04-07T12:00:00+10:00","updated":"2020-04-07T12:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-04-07:\/django-spa-infrastructure.html","summary":"<p>You're developing a web app with a Django REST backend and some sort of single page app frontend using React or Vue or something like that. There are many ways for you to run this app in production. There are a lot of choices that you need to make:<\/p>\n<ul>\n<li>Do \u2026<\/li><\/ul>","content":"<p>You're developing a web app with a Django REST backend and some sort of single page app frontend using React or Vue or something like that. There are many ways for you to run this app in production. There are a lot of choices that you need to make:<\/p>\n<ul>\n<li>Do you serve your frontend as a stand-alone static site or via Django views?<\/li>\n<li>Do you put the backend and frontend on different subdomains?<\/li>\n<li>Do you deploy the backend and frontend separately, or together?<\/li>\n<\/ul>\n<p>How do you choose? What is \"the right way\"?<\/p>\n<p>Well, the bad news is that there is no \"right way\" to do this and there are a lot of different trade-offs to consider. The good news is that I've compiled three different options with their pros and cons.<\/p>\n<h3>Option 1 - Cram it all into Django<\/h3>\n<p>This is the \"default\" approach, where you have a Django site and you just add React to it. All your HTML is served via Django views, and all your JavaScript and CSS is bundled by Django and served as static files. All your code, frontend and backend, is in one Git repository. You serve the app from a single domain like www.myapp.com.<\/p>\n<p>When you deploy your code using this setup, you will need to:<\/p>\n<ul>\n<li>Use <a href=\"https:\/\/webpack.js.org\">webpack<\/a>, or something <a href=\"https:\/\/www.google.com\/search?q=webpack+alternatives\">similar<\/a>, to build your JavaScript and CSS assets and put them into a Django static files directory<\/li>\n<li>Deploy Django like you usually would<\/li>\n<\/ul>\n<p>You will need to use a setup like <a href=\"https:\/\/pascalw.me\/blog\/2020\/04\/19\/webpack-django.html\">this<\/a> or <a href=\"https:\/\/github.com\/owais\/django-webpack-loader\">django-webpack-loader<\/a> to integrate Webpack's build assets with Django's staticfiles system and templates. Other than that, it's a vanilla Django deployment.<\/p>\n<p>The pros are:<\/p>\n<ul>\n<li><strong>Simplest infrastructure.<\/strong> Other than setting up django-webpack-loader and adding a Webpack build to the start of your deployment process, there's nothing else you need to do to your production infrastructure. Nothing extra to set up, pay for, configure, debug or tear your hair out over.<\/li>\n<li><strong>Cross-cutting changes.<\/strong> If you need to make a change that affects both your frontend and backend, then you can do it all in one Git commit and get your changes into production using a single deployment.<\/li>\n<li><strong>Tighter integration.<\/strong> With this setup you can use Django's views to pass context data from the backend to the frontend via templates. In addition, you can do server side rendering (with additional messing around with NodeJS).<\/li>\n<\/ul>\n<p>The cons are:<\/p>\n<ul>\n<li><strong>Single deployment for frontend and backend.<\/strong> Often you want to just deploy a small CSS or content change to the frontend, or a backend-only change. With this setup, you are forced to always deploy the backend and the frontend together. This means that you need to wait for the frontend to build, even if you didn't make any frontend changes! Even worse, a broken test, or linter error in the <em>other codebase<\/em> can fail a deployment, if you're using continuous integration practices. You don't want your database migration deployment to fail just because someone forgot to use semicolons in their JavaScript.<\/li>\n<li><strong>Tangled tech stack.<\/strong> Backend devs will need to know a little React, and frontend devs will need to know a little Django for this system to work.<\/li>\n<li><strong>Tricksy django-webpack-loader.<\/strong> Setting up the integration between Webpack and Django has been a painful process for me in the past. I don't remember why, I just remember pain. Truthfully, every option on this list will involve you wanting to throw your computer out of a window at some point, and this one is no exception.<\/li>\n<\/ul>\n<p>Choose this when:<\/p>\n<ul>\n<li>You want to keep your infrastructure simple<\/li>\n<li>You don't care about deployment times<\/li>\n<li>You typically deploy the frontend and backend together<\/li>\n<li>You need a tight integration between the frontend and backend (eg. data passing, server-side rendering)<\/li>\n<\/ul>\n<h3>Option 2 - Completely separate infrastructure<\/h3>\n<p>This is an approach that has become more popular over the last several years. In this setup you have two separate codebases, one for the frontend and one for the backend, each with their own Git repository.<\/p>\n<p>The frontend is deployed as a \"static site\" of just HTML CSS and JavaScript assets. It is hosted separately to Django, in an <a href=\"https:\/\/docs.aws.amazon.com\/AmazonS3\/latest\/dev\/WebsiteHosting.html\">AWS S3 bucket<\/a>, <a href=\"https:\/\/www.netlify.com\/\">Netlify<\/a>, or something similar. The frontend is built, tested and deployed independently of the backend. The frontend gets data from the backend soley through REST API calls.<\/p>\n<p>The backend is a Django REST API with no HTML views (other than the admin pages), and hosts no static content (other than what's needed for the admin). It is built, tested and deployed independently of the frontend.<\/p>\n<p>Importantly, since the frontend and backend are on different servers, they will also have different domain names. The backend might live on something like api.myapp.com and the frontend on www.myapp.com.<\/p>\n<p>The pros are:<\/p>\n<ul>\n<li><strong>Independent deployments.<\/strong> No waiting on the backend to deploy the frontend and vice versa.<\/li>\n<li><strong>Separation of concerns.<\/strong> Backend developers only need to think about the API, not views or CSS. Frontend developers only need to think in terms of the API presented by the backend, not the internal workings of Django. You <em>can<\/em> achieve this using option 1, but this method enforces it more strictly.<\/li>\n<li><strong>If the backend goes down, the frontend still works.<\/strong> Your users will still experience errors, but the site won't appear as broken.<\/li>\n<li><strong>Security permissions can be split up.<\/strong> You can split up who is allowed to deploy the frontend vs the backend, typically meaning more people will have the power to deploy, making your team more productive.<\/li>\n<\/ul>\n<p>The cons are:<\/p>\n<ul>\n<li><strong>More infrastructure.<\/strong> You will need to set up and maintain the static site plus an extra deployment process, which is more work, more complexity.<\/li>\n<li><strong>Cross-domain fuckery.<\/strong> You run into several problems because your frontend is on a different subdomain to your backend. You need to do some extra configuration of your Django settings to allow the frontend to talk to the backend properly. It's a security thing apparently. If you don't fix this you can have issues with making API requests to the backend, receiving cookies, and stuff like that. I don't understand it super well. I don't <em>want<\/em> to understand it super well. I have better shit to do than figure out the correct value of SESSION_COOKIE_DOMAIN, CORS_ORIGIN_REGEX_WHITELIST and friends. Even worse, cross-domain issues do not crop up on your local machine, because everything is served from localhost, so you need to deploy your configuration before you know if you got it right.<\/li>\n<\/ul>\n<p>Here are some cross domain Django settings that I hope you never need to think about:<\/p>\n<ul>\n<li>SESSION_COOKIE_DOMAIN<\/li>\n<li>CSRF_COOKIE_DOMAIN<\/li>\n<li>CSRF_TRUSTED_ORIGINS<\/li>\n<li>CORS_ORIGIN_ALLOW_ALL<\/li>\n<li>CORS_ALLOW_CREDENTIALS<\/li>\n<li>CORS_ORIGIN_REGEX_WHITELIST<\/li>\n<\/ul>\n<p>Choose this when:<\/p>\n<ul>\n<li>You have separate dedicated frontend and backend developers<\/li>\n<li>You want to deploy the backend and frontend separately<\/li>\n<li>You want to <em>completely<\/em> decouple your backend and frontend infrastructure<\/li>\n<li>You don't mind a little more operational complexity and configuration<\/li>\n<\/ul>\n<h3>Option 3 - One server, separate deployments<\/h3>\n<p>This approach is an attempted fusion of options 1 and 2. The idea is to still deploy the frontend as a separate static site, but you deploy everything to one server, under a single domain name:<\/p>\n<ul>\n<li>You have two separate codebases for the backend and frontend respectively<\/li>\n<li>Both codebases are deployed indepdently, but to the same server<\/li>\n<li>Both codebases are hosted on a single domain, like wwww.myapp.com<\/li>\n<\/ul>\n<p>You manage this by using a webserver, like NGINX, which handles all incoming requests. Requests to the URL path \/api\/ get sent to the WSGI server which runs your Django app (traditional reverse-proxy setup), while all other requests are sent to the frontend, which is set up as a static site and served from the filesystem (eg. \/var\/www\/).<\/p>\n<p>The pros are:<\/p>\n<ul>\n<li><strong>Most of the benefits of Option 2.<\/strong> Separation of concerns and independent deployments are still possible.<\/li>\n<li><strong>No \"cross-domain fuckery\".<\/strong> Since all requests are served from one domain, you shouldn't need to mess around with all those horrible cross-domain settings in Django.<\/li>\n<\/ul>\n<p>The cons are:<\/p>\n<ul>\n<li><strong>More infrastructure.<\/strong> This setup is still more complex than the \"Cram it all into Django\" option.<\/li>\n<li><strong>Requires control over host webserver.<\/strong> You need to be able to install and configure NGINX, deploy files to the filesystem etc. to get this done. This is straightforward if you're using a typical cloud virtual machine, but might be more tricky if you're using something like Heroku (not sure).<\/li>\n<\/ul>\n<p>Choose this when:<\/p>\n<ul>\n<li>You want to split up frontend and backend, but you don't need completely separate infrastructure<\/li>\n<li>You have sufficient control over your host webserver<\/li>\n<\/ul>\n<p>I'll be honest here. I actually have never tried option 3 (I've used 1 + 2 before). I thought it up when replying to a Reddit post. I think it'll work though. Good luck!<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"How to restart Celery on file change","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/restart-celery-on-file-change.html","rel":"alternate"}},"published":"2020-04-07T12:00:00+10:00","updated":"2020-04-07T12:00:00+10:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-04-07:\/restart-celery-on-file-change.html","summary":"<p>I use Celery and Django together a lot. My biggest pain when doing local development with Celery is that the worker process won't restart when I change my code. Django's <code>runserver<\/code> restarts on code change, why can't Celery? How can you set up your dev envrionent to force Celery to \u2026<\/p>","content":"<p>I use Celery and Django together a lot. My biggest pain when doing local development with Celery is that the worker process won't restart when I change my code. Django's <code>runserver<\/code> restarts on code change, why can't Celery? How can you set up your dev envrionent to force Celery to restart on file change?<\/p>\n<p><a href=\"https:\/\/github.com\/gorakhargosh\/watchdog\">Watchdog<\/a> is a nifty little (cross-platform?) Python library that watches for filesystem changes. We can use Watchdog to restart <em>anything<\/em> on file change.<\/p>\n<p>First, you need to install it in your local Python environment:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>pip install watchdog<span class=\"o\">[<\/span>watchmedo<span class=\"o\">]<\/span>\n<\/code><\/pre><\/div>\n\n<p>Let's say you normally start Celery like this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>celery worker --broker redis:\/\/localhost:6379 --app myapp\n<\/code><\/pre><\/div>\n\n<p>Now you can start it like this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>watchmedo <span class=\"se\">\\<\/span>\n    auto-restart <span class=\"se\">\\<\/span>\n    --directory .\/my-code\/ <span class=\"se\">\\<\/span>\n    --recursive <span class=\"se\">\\<\/span>\n    --pattern <span class=\"s1\">&#39;*.py&#39;<\/span> <span class=\"se\">\\<\/span>\n    -- <span class=\"se\">\\<\/span>\n    celery worker --broker redis:\/\/localhost:6379 --app myapp\n<\/code><\/pre><\/div>\n\n<p>That's it! Watchdog will restart the process on file change. If you like, you can specify:<\/p>\n<ul>\n<li>multiple code directories using --directory<\/li>\n<li>different file patterns using --pattern<\/li>\n<\/ul>","category":{"@attributes":{"term":"Django"}}},{"title":"How to deploy Django migrations","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/deploy-django-migrations.html","rel":"alternate"}},"published":"2020-04-04T12:00:00+11:00","updated":"2020-04-04T12:00:00+11:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-04-04:\/deploy-django-migrations.html","summary":"<p>You've started learning Django, you've created a new Django app and you've deployed it to a Linux webserver in the cloud somewhere. It's all set up and running nicely. Now you want to make some more changes and you need to update your models.<\/p>\n<p>How do you deploy those model \u2026<\/p>","content":"<p>You've started learning Django, you've created a new Django app and you've deployed it to a Linux webserver in the cloud somewhere. It's all set up and running nicely. Now you want to make some more changes and you need to update your models.<\/p>\n<p>How do you deploy those model changes? How do you get those changes into your production database?<\/p>\n<p>First I'll show you a simple way to run your migrations on a deployed Django app with a worked example, then I'll discuss some more advanced considerations.<\/p>\n<h3>Simple method<\/h3>\n<p>This simple method is how I like to run migrations. It's for when your web app:<\/p>\n<ul>\n<li>is a personal project<\/li>\n<li>has low traffic<\/li>\n<li>is only used by internal staff members<\/li>\n<\/ul>\n<p>Basically any situation where a few seconds of downtime isn't that important.<\/p>\n<h4>Update your model<\/h4>\n<p>First you need to make the changes you want to your model class.<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"k\">class<\/span> <span class=\"nc\">Person<\/span><span class=\"p\">(<\/span><span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">Model<\/span><span class=\"p\">):<\/span>\n    <span class=\"sd\">&quot;&quot;&quot;A human person&quot;&quot;&quot;<\/span>\n    <span class=\"n\">name<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">CharField<\/span><span class=\"p\">(<\/span><span class=\"n\">max_length<\/span><span class=\"o\">=<\/span><span class=\"mi\">128<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">height<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">FloatField<\/span><span class=\"p\">()<\/span>\n    <span class=\"c1\"># Add new attribute &quot;weight&quot;<\/span>\n    <span class=\"n\">weight<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">FloatField<\/span><span class=\"p\">()<\/span>\n<\/code><\/pre><\/div>\n\n<h4>Create the migration script locally<\/h4>\n<p>Once that is done, you want to use Django's command-line managment script <code>makemigrations<\/code> to auto-generate the migrations script.<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>.\/manage.py makemigrations\n<\/code><\/pre><\/div>\n\n<p>You should see a new file in your app's migrations folder. It'll have some whacky name like <code>0002_auto_20170219_2310.py<\/code>. If you're using Git, don't forget to commit this file.<\/p>\n<h4>Run the migration script locally<\/h4>\n<p>The <code>makemigrations<\/code> command only generates a script which applies your models changes to the database. To actually run that code and apply the changes, you need to run the <code>migrate<\/code> script:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>.\/manage.py migrate\n<\/code><\/pre><\/div>\n\n<h4>Check nothing broke<\/h4>\n<p>After you've done that, you should do some testing to make sure that the migrations actually worked. Check the admin panel to see that the model has changed in your local database, test out your app to see that you haven't broken any existing functionality. If you've got automated tests, run them. Once you're happy that it's all good, move on.<\/p>\n<h4>Deploy the migrations<\/h4>\n<p>Now that you've generated your migration script it's time to apply it to the production database:<\/p>\n<ul>\n<li>Copy all your new code onto the server. Ideally instead of just picking single files, just copy all the .py files in your project to make sure you didn't miss anything<\/li>\n<li>Stop your WSGI server<\/li>\n<li>Delete all of the old code on the server, including any .pyc files<\/li>\n<li>Move the new code to where the old code was<\/li>\n<li>Apply your migrations with <code>.\/manage.py migrate<\/code><\/li>\n<li>Start your WSGI server again<\/li>\n<\/ul>\n<h4>Why delete all the old code?<\/h4>\n<p>It might seem scary deleting all your deployed code and replacing it, but the alternative of just uploading a few files is even more risky. You could miss:<\/p>\n<ul>\n<li>The auto-generated migration file<\/li>\n<li>The model file that you changed<\/li>\n<li>Any other code that you've updated which depends on the updated model<\/li>\n<\/ul>\n<p>It's best to just nuke everything and start from scratch. This will ensure that your production code stays the same as your development code.<\/p>\n<h4>Is this the best way to do it?<\/h4>\n<p>The good thing about this method is that you don't have to worry about keeping your migrations backwards compatible. Sometimes your model changes will break your old code, but not your new code, like when you remove a field from a Django model. This method will keep you from running into that issue.<\/p>\n<p>BUT, following these steps will take your site down for a few seconds, which is fine for a lot of cases, but is bad if any downtime is unnacceptable.<\/p>\n<p>Websites that need to always stay up usually use a method called <a href=\"https:\/\/rollout.io\/blog\/blue-green-deployment\/\">\"blue-green\" deployments<\/a>, where there are many severs running at once.<\/p>\n<p>If you are doing blue-green deployments, this method will not work, and you will need to construct and deploy <a href=\"https:\/\/gist.github.com\/majackson\/493c3d6d4476914ca9da63f84247407b\">backwards-compatible migrations<\/a>.<\/p>\n<p>Don't invent problems for yourself though, keep your process simple if you can.<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"Fix long running tasks in Django views","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/offline-tasks.html","rel":"alternate"}},"published":"2020-04-02T12:00:00+11:00","updated":"2020-04-02T12:00:00+11:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-04-02:\/offline-tasks.html","summary":"<p>What do you do if you have a Django view that runs too slow? Slow views are a bad user experience. Users hate waiting. Even worse, if the view takes too long to return a response, they will receive a \"408 Request Timeout\" error, completely ruining the website experience.<\/p>\n<p>Sometimes \u2026<\/p>","content":"<p>What do you do if you have a Django view that runs too slow? Slow views are a bad user experience. Users hate waiting. Even worse, if the view takes too long to return a response, they will receive a \"408 Request Timeout\" error, completely ruining the website experience.<\/p>\n<p>Sometimes you can fine tune your code and improve the performance enough to fix the slow runtime, but sometimes there's nothing you can do to make it faster. What do you do when your code looks like this?<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># views.py<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">my_slow_view<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">):<\/span>\n    <span class=\"sd\">&quot;&quot;&quot;<\/span>\n<span class=\"sd\">    Performs a long running task for the user (slow response time).<\/span>\n<span class=\"sd\">    &quot;&quot;&quot;<\/span>\n    <span class=\"n\">long_running_task<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"o\">.<\/span><span class=\"n\">user<\/span><span class=\"p\">)<\/span> <span class=\"c1\"># Takes 30s<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">HttpResponse<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;Your task is finished!&quot;<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n<p>This kind of situation can happen when you have to:<\/p>\n<ul>\n<li>call out to an external API which is slow<\/li>\n<li>do some computationally expensive data crunching<\/li>\n<li>make some slow database queries<\/li>\n<li>any combination of the above<\/li>\n<\/ul>\n<p>So how do you fix this problem? You can't make your <code>long_running_task<\/code> any faster - that's out of your control, so what can you do? The solution is to push the execution of your long running task <em>somewhere else<\/em>.<\/p>\n<h3>Somewhere else?<\/h3>\n<p>In Django, when your view function runs, everything is happening on one thread. That is to say, each line of code has to run one after the other. We want to push our long running code into a different thread so that the view doesn't have to wait for our task to finish before it can return a response. We want to do something like this:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># views.py<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">my_fast_view<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">):<\/span>\n    <span class=\"sd\">&quot;&quot;&quot;<\/span>\n<span class=\"sd\">    Performs a long running task for the user (quick response time).<\/span>\n<span class=\"sd\">    &quot;&quot;&quot;<\/span>\n    <span class=\"n\">run_offline<\/span><span class=\"p\">(<\/span><span class=\"n\">long_running_task<\/span><span class=\"p\">,<\/span> <span class=\"n\">request<\/span><span class=\"o\">.<\/span><span class=\"n\">user<\/span><span class=\"p\">)<\/span> <span class=\"c1\"># Takes 0.01s<\/span>\n    <span class=\"c1\"># ... runs for 30s somewhere else.<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">HttpResponse<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;Your will task be finished soon!&quot;<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n<p>This is a common problem and Django has a lot of tools that will provide this functionality. There's <a href=\"http:\/\/www.celeryproject.org\/\">Celery<\/a>, <a href=\"https:\/\/huey.readthedocs.io\/en\/latest\/django.html\">Huey<\/a>, <a href=\"https:\/\/github.com\/rq\/django-rq\">Django Redis Queue<\/a>. For most projects I recommend using <a href=\"https:\/\/django-q.readthedocs.io\/en\/latest\/\">Django Q<\/a>, for the reasons outlined in <a href=\"https:\/\/mattsegal.dev\/simple-scheduled-tasks.html\">this post<\/a>.<\/p>\n<h3>Setting up Django Q<\/h3>\n<p>To get started you need Django Q set up. You can skip past this section to the worked example below and do this later.<\/p>\n<p>The first thing to do is install the Django Q package alongside Django:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>pip install django-q\n<\/code><\/pre><\/div>\n\n<h4>Configure settings<\/h4>\n<p>Then we need to adjust our Django settings so that Django knows that it should use the Django Q app. We also need to configure Django Q to use the database as the task broker.<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># shop\/settings.py<\/span>\n\n<span class=\"c1\"># Add Django-Q to your installed apps.<\/span>\n<span class=\"n\">INSTALLED_APPS<\/span> <span class=\"o\">=<\/span> <span class=\"p\">[<\/span>\n    <span class=\"c1\"># ...<\/span>\n    <span class=\"s1\">&#39;django_q&#39;<\/span>\n<span class=\"p\">]<\/span>\n\n<span class=\"c1\"># Configure your Q cluster<\/span>\n<span class=\"c1\"># More details https:\/\/django-q.readthedocs.io\/en\/latest\/configure.html<\/span>\n<span class=\"n\">Q_CLUSTER<\/span> <span class=\"o\">=<\/span> <span class=\"p\">{<\/span>\n    <span class=\"s2\">&quot;name&quot;<\/span><span class=\"p\">:<\/span> <span class=\"s2\">&quot;shop&quot;<\/span><span class=\"p\">,<\/span>\n    <span class=\"s2\">&quot;orm&quot;<\/span><span class=\"p\">:<\/span> <span class=\"s2\">&quot;default&quot;<\/span><span class=\"p\">,<\/span>  <span class=\"c1\"># Use Django&#39;s ORM + database for broker<\/span>\n<span class=\"p\">}<\/span>\n<\/code><\/pre><\/div>\n\n<h4>Apply migrations<\/h4>\n<p>Once this is done, we need to run our database migrations to create the tables that Django Q needs:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>.\/manage.py migrate\n<\/code><\/pre><\/div>\n\n<h4>Run the task process<\/h4>\n<p>Finally, we need to run the Django Q process. This is the \"somewhere else\" where our long-running tasks will execute. If you don't run the qcluster management command, your offline tasks will never run. To get this process started, open a new terminal window start the Django Q cluster via the Django management script:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>.\/manage.py qcluster\n<\/code><\/pre><\/div>\n\n<h3>Worked example<\/h3>\n<p>Imagine you run a stock-trading website. Your user owns a bunch of stocks - like 60 different stocks. Sometimes they want to click a button to refresh all their stocks so they can see the latest prices. The problem is that you need to hit a 3rd party API to get the new prices. Say each API call takes 500ms, that's 30s of waiting!<\/p>\n<h4>Slow version<\/h4>\n<p>Consider the following Stock model:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># models.py<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Stock<\/span><span class=\"p\">(<\/span><span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">Model<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">code<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">CharField<\/span><span class=\"p\">(<\/span><span class=\"n\">max_length<\/span><span class=\"o\">=<\/span><span class=\"mi\">16<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">price<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">DecimalField<\/span><span class=\"p\">(<\/span><span class=\"n\">decimal_places<\/span><span class=\"o\">=<\/span><span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"n\">max_digits<\/span><span class=\"o\">=<\/span><span class=\"mi\">7<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">user<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">ForeignKey<\/span><span class=\"p\">(<\/span><span class=\"n\">User<\/span><span class=\"p\">,<\/span> <span class=\"n\">on_delete<\/span><span class=\"o\">=<\/span><span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">CASCADE<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n<p>... and this slow view:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># views.py<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">refresh_stocks_view<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">):<\/span>\n    <span class=\"sd\">&quot;&quot;&quot;<\/span>\n<span class=\"sd\">    Refreshes a user&#39;s stocks (slow version)<\/span>\n<span class=\"sd\">    &quot;&quot;&quot;<\/span>\n    <span class=\"n\">stocks<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Stocks<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span><span class=\"o\">.<\/span><span class=\"n\">filter<\/span><span class=\"p\">(<\/span><span class=\"n\">user<\/span><span class=\"o\">=<\/span><span class=\"n\">request<\/span><span class=\"o\">.<\/span><span class=\"n\">user<\/span><span class=\"p\">)<\/span>\n    <span class=\"c1\"># Go through all stocks and update prices, takes at least 30s<\/span>\n    <span class=\"k\">for<\/span> <span class=\"n\">stock<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">stocks<\/span><span class=\"p\">:<\/span>\n        <span class=\"n\">stock<\/span><span class=\"o\">.<\/span><span class=\"n\">price<\/span> <span class=\"o\">=<\/span> <span class=\"n\">some_api<\/span><span class=\"o\">.<\/span><span class=\"n\">fetch_price<\/span><span class=\"p\">(<\/span><span class=\"n\">stock<\/span><span class=\"o\">.<\/span><span class=\"n\">code<\/span><span class=\"p\">)<\/span>\n        <span class=\"n\">stock<\/span><span class=\"o\">.<\/span><span class=\"n\">save<\/span><span class=\"p\">()<\/span>\n\n    <span class=\"k\">return<\/span> <span class=\"n\">render<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">,<\/span> <span class=\"s1\">&#39;stocks.html&#39;<\/span><span class=\"p\">,<\/span> <span class=\"p\">{<\/span><span class=\"s1\">&#39;stocks&#39;<\/span><span class=\"p\">:<\/span> <span class=\"n\">stocks<\/span><span class=\"p\">})<\/span>\n<\/code><\/pre><\/div>\n\n<h4>Fast offline version<\/h4>\n<p>We can start by moving the slow code into a task function, which will run inside of Django Q. By convention, I like to put these into a <code>tasks.py<\/code> module:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># task.py<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">refresh_stocks_task<\/span><span class=\"p\">(<\/span><span class=\"n\">stock_ids<\/span><span class=\"p\">):<\/span>\n    <span class=\"sd\">&quot;&quot;&quot;<\/span>\n<span class=\"sd\">    Refreshes all stocks in `stock_ids`, a list of ids.<\/span>\n<span class=\"sd\">    &quot;&quot;&quot;<\/span>\n    <span class=\"n\">stocks<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Stocks<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span><span class=\"o\">.<\/span><span class=\"n\">filter<\/span><span class=\"p\">(<\/span><span class=\"n\">id__in<\/span><span class=\"o\">=<\/span><span class=\"n\">stock_ids<\/span><span class=\"p\">)<\/span><span class=\"o\">.<\/span><span class=\"n\">all<\/span><span class=\"p\">()<\/span>\n    <span class=\"c1\"># Go through all stocks and update prices, takes at least 30s<\/span>\n    <span class=\"k\">for<\/span> <span class=\"n\">stock<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">stocks<\/span><span class=\"p\">:<\/span>\n        <span class=\"n\">stock<\/span><span class=\"o\">.<\/span><span class=\"n\">price<\/span> <span class=\"o\">=<\/span> <span class=\"n\">some_api<\/span><span class=\"o\">.<\/span><span class=\"n\">fetch_price<\/span><span class=\"p\">(<\/span><span class=\"n\">stock<\/span><span class=\"o\">.<\/span><span class=\"n\">code<\/span><span class=\"p\">)<\/span>\n        <span class=\"n\">stock<\/span><span class=\"o\">.<\/span><span class=\"n\">save<\/span><span class=\"p\">()<\/span>\n<\/code><\/pre><\/div>\n\n<p>Note that the task function takes a list of ids (<code>stock_ids<\/code>) - why not a list of Stock objects? The reason is that when Django Q stores the task in the database, waiting for execution, the task arguments are serialized as a string (or something like that). A Django model cannot be serialized into a string, so we need to use the ids instead.<\/p>\n<p>Now that we've created the task function, we just need to call it from our view:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># views.py<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">django_q.tasks<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">async_task<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">.tasks<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">refresh_stocks_task<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">refresh_stocks_view<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">):<\/span>\n    <span class=\"sd\">&quot;&quot;&quot;<\/span>\n<span class=\"sd\">    Refreshes a user&#39;s stocks (fast version)<\/span>\n<span class=\"sd\">    &quot;&quot;&quot;<\/span>\n    <span class=\"n\">stocks<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Stocks<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span><span class=\"o\">.<\/span><span class=\"n\">filter<\/span><span class=\"p\">(<\/span><span class=\"n\">user<\/span><span class=\"o\">=<\/span><span class=\"n\">request<\/span><span class=\"o\">.<\/span><span class=\"n\">user<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">stock_ids<\/span> <span class=\"o\">=<\/span> <span class=\"n\">stocks<\/span><span class=\"o\">.<\/span><span class=\"n\">values_list<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;id&#39;<\/span><span class=\"p\">,<\/span> <span class=\"n\">flat<\/span><span class=\"o\">=<\/span><span class=\"kc\">True<\/span><span class=\"p\">)<\/span>\n    <span class=\"c1\"># Dispatch task to Django Q - runs in &lt;1s<\/span>\n    <span class=\"n\">async_task<\/span><span class=\"p\">(<\/span><span class=\"n\">refresh_stocks_task<\/span><span class=\"p\">,<\/span> <span class=\"n\">stock_ids<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">render<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">,<\/span> <span class=\"s1\">&#39;stocks.html&#39;<\/span><span class=\"p\">,<\/span> <span class=\"p\">{<\/span><span class=\"s1\">&#39;stocks&#39;<\/span><span class=\"p\">:<\/span> <span class=\"n\">stocks<\/span><span class=\"p\">})<\/span>\n<\/code><\/pre><\/div>\n\n<p>That's basically it, but there's one level of complexity we can add for a better user experience.<\/p>\n<h4>Loading state<\/h4>\n<p>In the slow version, the user submits a request, waits 30s and eventually gets a response back with the new stock prices. In the fast version, the user gets a response back much faster, but their stock data isn't updated yet! They'll have to wait 30s and refresh the page to get the latest data, but there's no indication that anything happened. We can add a loading state the Stocks model to help the user understand what is going on:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># models.py<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Stock<\/span><span class=\"p\">(<\/span><span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">Model<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">code<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">CharField<\/span><span class=\"p\">(<\/span><span class=\"n\">max_length<\/span><span class=\"o\">=<\/span><span class=\"mi\">16<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">price<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">DecimalField<\/span><span class=\"p\">(<\/span><span class=\"n\">decimal_places<\/span><span class=\"o\">=<\/span><span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"n\">max_digits<\/span><span class=\"o\">=<\/span><span class=\"mi\">7<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">user<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">ForeignKey<\/span><span class=\"p\">(<\/span><span class=\"n\">User<\/span><span class=\"p\">,<\/span> <span class=\"n\">on_delete<\/span><span class=\"o\">=<\/span><span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">CASCADE<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">is_loading<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">BooleanField<\/span><span class=\"p\">(<\/span><span class=\"n\">default<\/span><span class=\"o\">=<\/span><span class=\"kc\">False<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div>\n\n<p>Then in the view we can set all our pending Stocks to \"loading\":<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># views.py<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">django_q.tasks<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">async_task<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">.tasks<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">refresh_stocks_task<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">refresh_stocks_view<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">):<\/span>\n    <span class=\"sd\">&quot;&quot;&quot;<\/span>\n<span class=\"sd\">    Refreshes a user&#39;s stocks (fast version)<\/span>\n<span class=\"sd\">    &quot;&quot;&quot;<\/span>\n    <span class=\"n\">stocks<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Stocks<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span><span class=\"o\">.<\/span><span class=\"n\">filter<\/span><span class=\"p\">(<\/span><span class=\"n\">user<\/span><span class=\"o\">=<\/span><span class=\"n\">request<\/span><span class=\"o\">.<\/span><span class=\"n\">user<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">stock_ids<\/span> <span class=\"o\">=<\/span> <span class=\"n\">stocks<\/span><span class=\"o\">.<\/span><span class=\"n\">values_list<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;id&#39;<\/span><span class=\"p\">,<\/span> <span class=\"n\">flat<\/span><span class=\"o\">=<\/span><span class=\"kc\">True<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">stocks<\/span><span class=\"o\">.<\/span><span class=\"n\">update<\/span><span class=\"p\">(<\/span><span class=\"n\">is_loading<\/span><span class=\"o\">=<\/span><span class=\"kc\">True<\/span><span class=\"p\">)<\/span>\n    <span class=\"c1\"># Dispatch task to Django Q - runs in &lt;1s<\/span>\n    <span class=\"n\">async_task<\/span><span class=\"p\">(<\/span><span class=\"n\">refresh_stocks_task<\/span><span class=\"p\">,<\/span> <span class=\"n\">stock_ids<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">render<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">,<\/span> <span class=\"s1\">&#39;stocks.html&#39;<\/span><span class=\"p\">,<\/span> <span class=\"p\">{<\/span><span class=\"s1\">&#39;stocks&#39;<\/span><span class=\"p\">:<\/span> <span class=\"n\">stocks<\/span><span class=\"p\">})<\/span>\n<\/code><\/pre><\/div>\n\n<p>Finally, we can set the Stock state back to \"not loading\" when the new price is fetched:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># task.py<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">refresh_stocks_task<\/span><span class=\"p\">(<\/span><span class=\"n\">stock_ids<\/span><span class=\"p\">):<\/span>\n    <span class=\"sd\">&quot;&quot;&quot;<\/span>\n<span class=\"sd\">    Refreshes all stocks in `stock_ids`, a list of ids.<\/span>\n<span class=\"sd\">    &quot;&quot;&quot;<\/span>\n    <span class=\"n\">stocks<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Stocks<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span><span class=\"o\">.<\/span><span class=\"n\">filter<\/span><span class=\"p\">(<\/span><span class=\"n\">id__in<\/span><span class=\"o\">=<\/span><span class=\"n\">stock_ids<\/span><span class=\"p\">)<\/span><span class=\"o\">.<\/span><span class=\"n\">all<\/span><span class=\"p\">()<\/span>\n    <span class=\"c1\"># Go through all stocks and update prices, takes at least 30s<\/span>\n    <span class=\"k\">for<\/span> <span class=\"n\">stock<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">stocks<\/span><span class=\"p\">:<\/span>\n        <span class=\"n\">stock<\/span><span class=\"o\">.<\/span><span class=\"n\">price<\/span> <span class=\"o\">=<\/span> <span class=\"n\">some_api<\/span><span class=\"o\">.<\/span><span class=\"n\">fetch_price<\/span><span class=\"p\">(<\/span><span class=\"n\">stock<\/span><span class=\"o\">.<\/span><span class=\"n\">code<\/span><span class=\"p\">)<\/span>\n        <span class=\"n\">stock<\/span><span class=\"o\">.<\/span><span class=\"n\">is_loading<\/span> <span class=\"o\">=<\/span> <span class=\"kc\">False<\/span>\n        <span class=\"n\">stock<\/span><span class=\"o\">.<\/span><span class=\"n\">save<\/span><span class=\"p\">()<\/span>\n<\/code><\/pre><\/div>\n\n<p>Now the user will request a refresh, see that all of their stocks are loading, and when the new prices have been set the user will see them once they refresh the page again.<\/p>\n<p>That's it, hopefully you can now get started doing offline processing in Django. Enjoy!<\/p>","category":{"@attributes":{"term":"Django"}}},{"title":"Simple scheduled tasks with Django Q","link":{"@attributes":{"href":"https:\/\/mattsegal.dev\/simple-scheduled-tasks.html","rel":"alternate"}},"published":"2020-03-30T12:00:00+11:00","updated":"2020-03-30T12:00:00+11:00","author":{"name":"Matthew Segal"},"id":"tag:mattsegal.dev,2020-03-30:\/simple-scheduled-tasks.html","summary":"<p>How do you run some code once a day in Django, or every hour? This post will explain how to set up scheduled code execution in Django using Django-Q.<\/p>\n<p>There are a lot of reasons you might want to run code on a schedule. You may want to:<\/p>\n<ul>\n<li>Process a \u2026<\/li><\/ul>","content":"<p>How do you run some code once a day in Django, or every hour? This post will explain how to set up scheduled code execution in Django using Django-Q.<\/p>\n<p>There are a lot of reasons you might want to run code on a schedule. You may want to:<\/p>\n<ul>\n<li>Process a batch of data every night<\/li>\n<li>Send out a bunch of emails once a week<\/li>\n<li>Regularly scrape a website and store the results in the database<\/li>\n<\/ul>\n<p>If you're running a backend web service, you will need to do something like this eventually.<\/p>\n<p>When you ask around online for help with setting up a scheduler in Django, people will often point you to <a href=\"http:\/\/www.celeryproject.org\/\">Celery<\/a>. If you look at Celery's website:<\/p>\n<blockquote>\n<p>Celery is an asynchronous task queue\/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.<\/p>\n<\/blockquote>\n<p>Asynchronous what? Distributed? Sounds complicated. Do I need that? Celery is intimidating for beginners, and it happens to be pain in the ass to set up. If you happen to need Celery, then it's well worth the effort, but I believe that it's overkill for most people.<\/p>\n<p>The biggest stumbling block is that Celery requires that you set up some kind of \"<a href=\"http:\/\/docs.celeryproject.org\/en\/latest\/getting-started\/brokers\/\">broker<\/a>\", which is a program which keeps track of all the tasks that need to be done. You will need to install and run a program like <a href=\"https:\/\/redis.io\/\">Redis<\/a> or <a href=\"https:\/\/www.rabbitmq.com\/\">RabbitMQ<\/a> to run Celery, which makes getting started more complciated, and gives you more infrastructure to worry about.<\/p>\n<p>I think the best solution for beginners is <a href=\"https:\/\/django-q.readthedocs.io\/en\/latest\/\">Django-Q<\/a>. It's simpler to set up and run in production than Celery, and it is perfectly fine for basic scheduling tasks. Django-Q can use just your existing database as a broker, which means you don't have to set up any new infrastructure. If you find that you need to use a different broker later on, then you can swap out the database for something else.<\/p>\n<h2>Example project<\/h2>\n<p>The <a href=\"https:\/\/django-q.readthedocs.io\/en\/latest\/install.html\">Django-Q installation docs<\/a> are reasonably good, but if you're new to programming you might struggle to put all the pieces together. I've created a worked example to try to give you the full picture. You can check out the full code on <a href=\"https:\/\/github.com\/MattSegal\/devblog-examples\/tree\/master\/django-q-scheduling-example\">GitHub<\/a>.<\/p>\n<p>Let's say I have a Django app that is and online store which has a Discount model. This model keeps track of:<\/p>\n<ul>\n<li>when it was created (<code>created_at<\/code>)<\/li>\n<li>the amount that should be discounted (<code>amount<\/code>)<\/li>\n<\/ul>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\">#  discounts\/models.py<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Discount<\/span><span class=\"p\">(<\/span><span class=\"n\">model<\/span><span class=\"o\">.<\/span><span class=\"n\">Model<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">created_at<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">DateTimeField<\/span><span class=\"p\">(<\/span><span class=\"n\">default<\/span><span class=\"o\">=<\/span><span class=\"n\">timezone<\/span><span class=\"o\">.<\/span><span class=\"n\">now<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">amount<\/span> <span class=\"o\">=<\/span> <span class=\"n\">models<\/span><span class=\"o\">.<\/span><span class=\"n\">IntegerField<\/span><span class=\"p\">()<\/span>\n<\/code><\/pre><\/div>\n\n<p>And let's say that every minute I want to delete every discount that is older than a minute. It's a silly thing to do, but this is just an learning example. So how do we set up Django-Q to do this?<\/p>\n<div class=\"ui divider\" style=\"margin: 1.5em 0;\"><\/div>\n<form action=\"https:\/\/dev.us19.list-manage.com\/subscribe\/post?u=e7a1ec466f7bb1732dbd23fc7&amp;id=ec345473bd\" method=\"post\" name=\"mc-embedded-subscribe-form\" target=\"_blank\" style=\"text-align: center; padding-bottom: 1em;\" novalidate>\n  <h3 class=\"subscribe-cta\">Get alerted when I publish new blog posts<\/h3>\n  <div class=\"ui fluid action input subscribe\">\n    <input\n      type=\"email\"\n      value=\"\"\n      name=\"EMAIL\"\n      placeholder=\"Enter your email address\"\n    \/>\n    <button class=\"ui primary button\" type=\"submit\" name=\"subscribe\">\n      Subscribe\n    <\/button>\n  <\/div>\n  <div style=\"position: absolute; left: -5000px;\" aria-hidden=\"true\">\n    <input\n      type=\"text\"\n      name=\"b_e7a1ec466f7bb1732dbd23fc7_ec345473bd\"\n      tabindex=\"-1\"\n      value=\"\"\n    \/>\n  <\/div>\n<\/form>\n<div class=\"ui divider\" style=\"margin: 1.5em 0;\"><\/div>\n\n<h2>Install the package<\/h2>\n<p>First thing to do is install the Django-Q package:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>pip install django-q\n<\/code><\/pre><\/div>\n\n<h2>Configure settings<\/h2>\n<p>Then we need to adjust our Django settings so that Django knows that it should use the Django-Q app. We also need to configure Django-Q to use the database as the task broker.<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># shop\/settings.py<\/span>\n\n<span class=\"c1\"># Add Django-Q to your installed apps.<\/span>\n<span class=\"n\">INSTALLED_APPS<\/span> <span class=\"o\">=<\/span> <span class=\"p\">[<\/span>\n    <span class=\"c1\"># ...<\/span>\n    <span class=\"s1\">&#39;django_q&#39;<\/span>\n<span class=\"p\">]<\/span>\n\n<span class=\"c1\"># Configure your Q cluster<\/span>\n<span class=\"c1\"># More details https:\/\/django-q.readthedocs.io\/en\/latest\/configure.html<\/span>\n<span class=\"n\">Q_CLUSTER<\/span> <span class=\"o\">=<\/span> <span class=\"p\">{<\/span>\n    <span class=\"s2\">&quot;name&quot;<\/span><span class=\"p\">:<\/span> <span class=\"s2\">&quot;shop&quot;<\/span><span class=\"p\">,<\/span>\n    <span class=\"s2\">&quot;orm&quot;<\/span><span class=\"p\">:<\/span> <span class=\"s2\">&quot;default&quot;<\/span><span class=\"p\">,<\/span>  <span class=\"c1\"># Use Django&#39;s ORM + database for broker<\/span>\n<span class=\"p\">}<\/span>\n<\/code><\/pre><\/div>\n\n<h2>Apply migrations<\/h2>\n<p>Once this is done, we need to run our database migrations to create the tables that Django-Q needs:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>.\/manage.py migrate\n<\/code><\/pre><\/div>\n\n<h2>Create a task<\/h2>\n<p>Next we need to create the task function that will be called every minute. I've decided to put mine in a <code>tasks.py<\/code> module. You can see below that there's nothing special about this - just a plain old Python function.<\/p>\n<div class=\"highlight\"><pre><span><\/span><code><span class=\"c1\"># discounts\/tasks.py<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">delete_expired_discounts<\/span><span class=\"p\">():<\/span>\n    <span class=\"sd\">&quot;&quot;&quot;<\/span>\n<span class=\"sd\">    Deletes all Discounts that are more than a minute old<\/span>\n<span class=\"sd\">    &quot;&quot;&quot;<\/span>\n    <span class=\"n\">one_minute_ago<\/span> <span class=\"o\">=<\/span> <span class=\"n\">timezone<\/span><span class=\"o\">.<\/span><span class=\"n\">now<\/span><span class=\"p\">()<\/span> <span class=\"o\">-<\/span> <span class=\"n\">timezone<\/span><span class=\"o\">.<\/span><span class=\"n\">timedelta<\/span><span class=\"p\">(<\/span><span class=\"n\">minutes<\/span><span class=\"o\">=<\/span><span class=\"mi\">1<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">expired_discounts<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Discount<\/span><span class=\"o\">.<\/span><span class=\"n\">objects<\/span><span class=\"o\">.<\/span><span class=\"n\">filter<\/span><span class=\"p\">(<\/span>\n        <span class=\"n\">created_at__lte<\/span><span class=\"o\">=<\/span><span class=\"n\">one_minute_ago<\/span>\n    <span class=\"p\">)<\/span>\n    <span class=\"n\">expired_discounts<\/span><span class=\"o\">.<\/span><span class=\"n\">delete<\/span><span class=\"p\">()<\/span>\n<\/code><\/pre><\/div>\n\n<h2>Create a schedule<\/h2>\n<p>Now that we have a task ready to run, we need to add a scheduled task to the database. We can do this on the admin site at <code>\/admin\/django_q\/schedule\/add\/<\/code>, or we can create and save a Schedule instance (<a href=\"https:\/\/django-q.readthedocs.io\/en\/latest\/schedules.html\">docs here<\/a>) using the Django shell:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>.\/manage.py shell\nfrom django_q.models import Schedule\nSchedule.objects.create<span class=\"o\">(<\/span>\n    <span class=\"nv\">func<\/span><span class=\"o\">=<\/span><span class=\"s1\">&#39;discounts.tasks.delete_expired_discounts&#39;<\/span>,\n    <span class=\"nv\">minutes<\/span><span class=\"o\">=<\/span><span class=\"m\">1<\/span>,\n    <span class=\"nv\">repeats<\/span><span class=\"o\">=<\/span>-1\n<span class=\"o\">)<\/span>\n<\/code><\/pre><\/div>\n\n<h2>Run the scheduler<\/h2>\n<p>Finally, we need to run the Django-Q process. When using Django, you will usually have one process that is responsible for serving web requests and a separate one that takes care of processing tasks. During local development, these two processes are:<\/p>\n<ul>\n<li>web requests: <code>.\/manage.py runserver<\/code><\/li>\n<li>async tasks: <code>.\/manage.py qcluster<\/code><\/li>\n<\/ul>\n<p>So if you don't run the qcluster management command, the scheduled task will never run. To get this process started, open a new terminal window start the Django-Q cluster via the Django management script:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>.\/manage.py qcluster\n<\/code><\/pre><\/div>\n\n<p>Now you should see your scheduled task processing in the console output:<\/p>\n<div class=\"highlight\"><pre><span><\/span><code>12:54:18 [Q] INFO Process-1 created a task from schedule [2]\n<\/code><\/pre><\/div>\n\n<p>You can also see what's going on in the Django admin site at <code>\/admin\/django_q\/<\/code>.<\/p>\n<p>...and that's it! You can now run scheduled tasks in Django.<\/p>","category":{"@attributes":{"term":"Django"}}}]}