The Digital Cat - HTTPhttps://www.thedigitalcatonline.com/2021-08-22T10:00:00+00:00Adventures of a curious cat in the land of programmingFlask project setup: TDD, Docker, Postgres and more - Part 32020-07-07T13:00:00+01:002021-02-23T20:00:00+00:00Leonardo Giordanitag:www.thedigitalcatonline.com,2020-07-07:/blog/2020/07/07/flask-project-setup-tdd-docker-postgres-and-more-part-3/<p>A step-by-step tutorial on how to setup a Flask project with TDD, Docker and Postgres</p><p>In this series of posts I explore the development of a Flask project with a setup that is built with efficiency and tidiness in mind, using TDD, Docker and Postgres.</p><h2 id="catch-up-7f97">Catch-up<a class="headerlink" href="#catch-up-7f97" title="Permanent link">¶</a></h2><p>In the <a href="https://www.thedigitalcatonline.com/blog/2020/07/05/flask-project-setup-tdd-docker-postgres-and-more-part-1/">first</a> and <a href="https://www.thedigitalcatonline.com/blog/2020/07/06/flask-project-setup-tdd-docker-postgres-and-more-part-2/">second</a> posts I created a Flask project with a tidy setup, using Docker to run the development environment and the tests, and mapping important commands in a management script, so that the configuration can be in a single file and drive the whole system.</p><p>In this post I will show you how to easily create scenarios, that is databases created on the fly with custom data, so that it is possible to test queries in isolation, either with the Flask application or with the command line. I will also show you how to define a configuration for production and give some hints for the deployment.</p><h2 id="step-1---creating-scenarios-59b9">Step 1 - Creating scenarios<a class="headerlink" href="#step-1---creating-scenarios-59b9" title="Permanent link">¶</a></h2><p>The idea of scenarios is simple. Sometimes you need to investigate specific use cases for bugs, or maybe increase the performances of some database queries, and you might need to do this on a customised database. This is a scenario, a Python file that populates the database with a specific set of data and that allows you to run the application or the database shell on it.</p><p>Often the development database is a copy of the production one, maybe with sensitive data stripped to avoid leaking private information, and while this gives us a realistic case where to test queries (e.g. how does the query perform on 1 million lines?) it might not help during the initial investigations, where you need to have all the data in front of you to properly understand what happens. Whoever learned how joins work in relational databases understands what I mean here.</p><p>In principle, to create a scenario we just need to spin up an empty database and to run the scenario code against it. In practice, things are not much more complicated, but there are a couple of minor issues that we need to solve.</p><p>First, I am already running a database for the development and one for the testing. The second is ephemeral, but I decided to setup the project so that I can run the tests while the development database is up, and the way I did it was using port 5432 (the standard Postgres one) for development and 5433 for testing. Spinning up scenarios adds more databases to the equation. Clearly I do not expect to run 5 scenarios at the same time while running the development and the test databases, but I make myself a rule to make something generic as soon I do it for the third time.</p><p>This means that I won't create a database for a scenario on port 5434 and will instead look for a more generic solution. This is offered me by the Docker networking model, where I can map a container port to the host but avoid assigning the destination port, and it will be chosen randomly by Docker itself among the unprivileged ones. This means that I can create a Postgres container mapping port 5432 (the port in the container) and having Docker connect it to port 32838 in the host (for example). As long as the application knows which port to use this is absolutely the same as using port 5432.</p><p>Unfortunately the Docker interface is not extremely script-friendly when it comes to providing information and I have to parse the output a bit. Practically speaking, after I spin up the containers, I will run the command <code>docker-compose port db 5432</code> which will return a string like <code>0.0.0.0:32838</code>, and I will extract the port from it. Nothing major, but these are the (sometimes many) issues you face when you orchestrate different systems together.</p><p>The new management script is</p><div class="code"><div class="title"><code>manage.py</code></div><div class="content"><div class="highlight"><pre><span class="ch">#! /usr/bin/env python</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">signal</span>
<span class="kn">import</span> <span class="nn">subprocess</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="kn">import</span> <span class="nn">shutil</span>
<span class="kn">import</span> <span class="nn">click</span>
<span class="kn">import</span> <span class="nn">psycopg2</span>
<span class="kn">from</span> <span class="nn">psycopg2.extensions</span> <span class="kn">import</span> <span class="n">ISOLATION_LEVEL_AUTOCOMMIT</span>
<span class="c1"># Ensure an environment variable exists and has a value</span>
<span class="k">def</span> <span class="nf">setenv</span><span class="p">(</span><span class="n">variable</span><span class="p">,</span> <span class="n">default</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="n">variable</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="n">variable</span><span class="p">,</span> <span class="n">default</span><span class="p">)</span>
<span class="n">setenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">,</span> <span class="s2">"development"</span><span class="p">)</span>
<span class="n">APPLICATION_CONFIG_PATH</span> <span class="o">=</span> <span class="s2">"config"</span>
<span class="n">DOCKER_PATH</span> <span class="o">=</span> <span class="s2">"docker"</span>
<span class="k">def</span> <span class="nf">app_config_file</span><span class="p">(</span><span class="n">config</span><span class="p">):</span>
<span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">APPLICATION_CONFIG_PATH</span><span class="p">,</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">config</span><span class="si">}</span><span class="s2">.json"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">docker_compose_file</span><span class="p">(</span><span class="n">config</span><span class="p">):</span>
<span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">DOCKER_PATH</span><span class="p">,</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">config</span><span class="si">}</span><span class="s2">.yml"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">configure_app</span><span class="p">(</span><span class="n">config</span><span class="p">):</span>
<span class="c1"># Read configuration from the relative JSON file</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">app_config_file</span><span class="p">(</span><span class="n">config</span><span class="p">))</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">config_data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="c1"># Convert the config into a usable Python dictionary</span>
<span class="n">config_data</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">((</span><span class="n">i</span><span class="p">[</span><span class="s2">"name"</span><span class="p">],</span> <span class="n">i</span><span class="p">[</span><span class="s2">"value"</span><span class="p">])</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">config_data</span><span class="p">)</span>
<span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">config_data</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="n">setenv</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">group</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">cli</span><span class="p">():</span>
<span class="k">pass</span>
<span class="nd">@cli</span><span class="o">.</span><span class="n">command</span><span class="p">(</span><span class="n">context_settings</span><span class="o">=</span><span class="p">{</span><span class="s2">"ignore_unknown_options"</span><span class="p">:</span> <span class="kc">True</span><span class="p">})</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">argument</span><span class="p">(</span><span class="s2">"subcommand"</span><span class="p">,</span> <span class="n">nargs</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="n">click</span><span class="o">.</span><span class="n">Path</span><span class="p">())</span>
<span class="k">def</span> <span class="nf">flask</span><span class="p">(</span><span class="n">subcommand</span><span class="p">):</span>
<span class="n">configure_app</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">))</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"flask"</span><span class="p">]</span> <span class="o">+</span> <span class="nb">list</span><span class="p">(</span><span class="n">subcommand</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">KeyboardInterrupt</span><span class="p">:</span>
<span class="n">p</span><span class="o">.</span><span class="n">send_signal</span><span class="p">(</span><span class="n">signal</span><span class="o">.</span><span class="n">SIGINT</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">docker_compose_cmdline</span><span class="p">(</span><span class="n">commands_string</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="n">config</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">)</span>
<span class="n">configure_app</span><span class="p">(</span><span class="n">config</span><span class="p">)</span>
<span class="n">compose_file</span> <span class="o">=</span> <span class="n">docker_compose_file</span><span class="p">(</span><span class="n">config</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isfile</span><span class="p">(</span><span class="n">compose_file</span><span class="p">):</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">"The file </span><span class="si">{</span><span class="n">compose_file</span><span class="si">}</span><span class="s2"> does not exist"</span><span class="p">)</span>
<span class="n">command_line</span> <span class="o">=</span> <span class="p">[</span>
<span class="s2">"docker-compose"</span><span class="p">,</span>
<span class="s2">"-p"</span><span class="p">,</span>
<span class="n">config</span><span class="p">,</span>
<span class="s2">"-f"</span><span class="p">,</span>
<span class="n">compose_file</span><span class="p">,</span>
<span class="p">]</span>
<span class="k">if</span> <span class="n">commands_string</span><span class="p">:</span>
<span class="n">command_line</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">commands_string</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">" "</span><span class="p">))</span>
<span class="k">return</span> <span class="n">command_line</span>
<span class="nd">@cli</span><span class="o">.</span><span class="n">command</span><span class="p">(</span><span class="n">context_settings</span><span class="o">=</span><span class="p">{</span><span class="s2">"ignore_unknown_options"</span><span class="p">:</span> <span class="kc">True</span><span class="p">})</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">argument</span><span class="p">(</span><span class="s2">"subcommand"</span><span class="p">,</span> <span class="n">nargs</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="n">click</span><span class="o">.</span><span class="n">Path</span><span class="p">())</span>
<span class="k">def</span> <span class="nf">compose</span><span class="p">(</span><span class="n">subcommand</span><span class="p">):</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="n">docker_compose_cmdline</span><span class="p">()</span> <span class="o">+</span> <span class="nb">list</span><span class="p">(</span><span class="n">subcommand</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">KeyboardInterrupt</span><span class="p">:</span>
<span class="n">p</span><span class="o">.</span><span class="n">send_signal</span><span class="p">(</span><span class="n">signal</span><span class="o">.</span><span class="n">SIGINT</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">run_sql</span><span class="p">(</span><span class="n">statements</span><span class="p">):</span>
<span class="n">conn</span> <span class="o">=</span> <span class="n">psycopg2</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span>
<span class="n">dbname</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"POSTGRES_DB"</span><span class="p">),</span>
<span class="n">user</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"POSTGRES_USER"</span><span class="p">),</span>
<span class="n">password</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"POSTGRES_PASSWORD"</span><span class="p">),</span>
<span class="n">host</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"POSTGRES_HOSTNAME"</span><span class="p">),</span>
<span class="n">port</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"POSTGRES_PORT"</span><span class="p">),</span>
<span class="p">)</span>
<span class="n">conn</span><span class="o">.</span><span class="n">set_isolation_level</span><span class="p">(</span><span class="n">ISOLATION_LEVEL_AUTOCOMMIT</span><span class="p">)</span>
<span class="n">cursor</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
<span class="k">for</span> <span class="n">statement</span> <span class="ow">in</span> <span class="n">statements</span><span class="p">:</span>
<span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">statement</span><span class="p">)</span>
<span class="n">cursor</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">wait_for_logs</span><span class="p">(</span><span class="n">cmdline</span><span class="p">,</span> <span class="n">message</span><span class="p">):</span>
<span class="n">logs</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">check_output</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="k">while</span> <span class="n">message</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">logs</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">"utf-8"</span><span class="p">):</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span>
<span class="n">logs</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">check_output</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="nd">@cli</span><span class="o">.</span><span class="n">command</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">create_initial_db</span><span class="p">():</span>
<span class="n">configure_app</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">))</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">run_sql</span><span class="p">([</span><span class="sa">f</span><span class="s2">"CREATE DATABASE </span><span class="si">{</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'APPLICATION_DB'</span><span class="p">)</span><span class="si">}</span><span class="s2">"</span><span class="p">])</span>
<span class="k">except</span> <span class="n">psycopg2</span><span class="o">.</span><span class="n">errors</span><span class="o">.</span><span class="n">DuplicateDatabase</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">"The database </span><span class="si">{</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'APPLICATION_DB'</span><span class="p">)</span><span class="si">}</span><span class="s2"> already exists and will not be recreated"</span>
<span class="p">)</span>
<span class="nd">@cli</span><span class="o">.</span><span class="n">command</span><span class="p">()</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">argument</span><span class="p">(</span><span class="s2">"filenames"</span><span class="p">,</span> <span class="n">nargs</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test</span><span class="p">(</span><span class="n">filenames</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"testing"</span>
<span class="n">configure_app</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">))</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="n">docker_compose_cmdline</span><span class="p">(</span><span class="s2">"up -d"</span><span class="p">)</span>
<span class="n">subprocess</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="n">docker_compose_cmdline</span><span class="p">(</span><span class="s2">"logs db"</span><span class="p">)</span>
<span class="n">wait_for_logs</span><span class="p">(</span><span class="n">cmdline</span><span class="p">,</span> <span class="s2">"ready to accept connections"</span><span class="p">)</span>
<span class="n">run_sql</span><span class="p">([</span><span class="sa">f</span><span class="s2">"CREATE DATABASE </span><span class="si">{</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'APPLICATION_DB'</span><span class="p">)</span><span class="si">}</span><span class="s2">"</span><span class="p">])</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"pytest"</span><span class="p">,</span> <span class="s2">"-svv"</span><span class="p">,</span> <span class="s2">"--cov=application"</span><span class="p">,</span> <span class="s2">"--cov-report=term-missing"</span><span class="p">]</span>
<span class="n">cmdline</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">filenames</span><span class="p">)</span>
<span class="n">subprocess</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="n">docker_compose_cmdline</span><span class="p">(</span><span class="s2">"down"</span><span class="p">)</span>
<span class="n">subprocess</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="nd">@cli</span><span class="o">.</span><span class="n">group</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">scenario</span><span class="p">():</span>
<span class="k">pass</span>
<span class="nd">@scenario</span><span class="o">.</span><span class="n">command</span><span class="p">()</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">argument</span><span class="p">(</span><span class="s2">"name"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">up</span><span class="p">(</span><span class="n">name</span><span class="p">):</span> <span class="callout">1</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">]</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"scenario_</span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">"</span>
<span class="n">config</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">)</span>
<span class="n">scenario_config_source_file</span> <span class="o">=</span> <span class="n">app_config_file</span><span class="p">(</span><span class="s2">"scenario"</span><span class="p">)</span>
<span class="n">scenario_config_file</span> <span class="o">=</span> <span class="n">app_config_file</span><span class="p">(</span><span class="n">config</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isfile</span><span class="p">(</span><span class="n">scenario_config_source_file</span><span class="p">):</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">"File </span><span class="si">{</span><span class="n">scenario_config_source_file</span><span class="si">}</span><span class="s2"> doesn't exist"</span><span class="p">)</span>
<span class="n">shutil</span><span class="o">.</span><span class="n">copy</span><span class="p">(</span><span class="n">scenario_config_source_file</span><span class="p">,</span> <span class="n">scenario_config_file</span><span class="p">)</span> <span class="callout">3</span>
<span class="n">scenario_docker_source_file</span> <span class="o">=</span> <span class="n">docker_compose_file</span><span class="p">(</span><span class="s2">"scenario"</span><span class="p">)</span>
<span class="n">scenario_docker_file</span> <span class="o">=</span> <span class="n">docker_compose_file</span><span class="p">(</span><span class="n">config</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isfile</span><span class="p">(</span><span class="n">scenario_docker_source_file</span><span class="p">):</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">"File </span><span class="si">{</span><span class="n">scenario_docker_source_file</span><span class="si">}</span><span class="s2"> doesn't exist"</span><span class="p">)</span>
<span class="n">shutil</span><span class="o">.</span><span class="n">copy</span><span class="p">(</span><span class="n">docker_compose_file</span><span class="p">(</span><span class="s2">"scenario"</span><span class="p">),</span> <span class="n">scenario_docker_file</span><span class="p">)</span> <span class="callout">4</span>
<span class="n">configure_app</span><span class="p">(</span><span class="sa">f</span><span class="s2">"scenario_</span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="n">docker_compose_cmdline</span><span class="p">(</span><span class="s2">"up -d"</span><span class="p">)</span> <span class="callout">5</span>
<span class="n">subprocess</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="n">docker_compose_cmdline</span><span class="p">(</span><span class="s2">"logs db"</span><span class="p">)</span>
<span class="n">wait_for_logs</span><span class="p">(</span><span class="n">cmdline</span><span class="p">,</span> <span class="s2">"ready to accept connections"</span><span class="p">)</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="n">docker_compose_cmdline</span><span class="p">(</span><span class="s2">"port db 5432"</span><span class="p">)</span> <span class="callout">6</span>
<span class="n">out</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">check_output</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">port</span> <span class="o">=</span> <span class="n">out</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">"utf-8"</span><span class="p">)</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">""</span><span class="p">)</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">":"</span><span class="p">)[</span><span class="mi">1</span><span class="p">]</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"POSTGRES_PORT"</span><span class="p">]</span> <span class="o">=</span> <span class="n">port</span>
<span class="n">run_sql</span><span class="p">([</span><span class="sa">f</span><span class="s2">"CREATE DATABASE </span><span class="si">{</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'APPLICATION_DB'</span><span class="p">)</span><span class="si">}</span><span class="s2">"</span><span class="p">])</span>
<span class="n">scenario_module</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"scenarios.</span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">"</span>
<span class="n">scenario_file</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s2">"scenarios"</span><span class="p">,</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">.py"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isfile</span><span class="p">(</span><span class="n">scenario_file</span><span class="p">):</span> <span class="callout">7</span>
<span class="kn">import</span> <span class="nn">importlib</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"APPLICATION_SCENARIO_NAME"</span><span class="p">]</span> <span class="o">=</span> <span class="n">name</span>
<span class="n">scenario</span> <span class="o">=</span> <span class="n">importlib</span><span class="o">.</span><span class="n">import_module</span><span class="p">(</span><span class="n">scenario_module</span><span class="p">)</span>
<span class="n">scenario</span><span class="o">.</span><span class="n">run</span><span class="p">()</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="s2">" "</span><span class="o">.</span><span class="n">join</span><span class="p">(</span> <span class="callout">8</span>
<span class="n">docker_compose_cmdline</span><span class="p">(</span>
<span class="s2">"exec db psql -U </span><span class="si">{}</span><span class="s2"> -d </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
<span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"POSTGRES_USER"</span><span class="p">),</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_DB"</span><span class="p">)</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"Your scenario is ready. If you want to open a SQL shell run"</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="nd">@scenario</span><span class="o">.</span><span class="n">command</span><span class="p">()</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">argument</span><span class="p">(</span><span class="s2">"name"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">down</span><span class="p">(</span><span class="n">name</span><span class="p">):</span> <span class="callout">2</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">]</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"scenario_</span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">"</span>
<span class="n">config</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">)</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="n">docker_compose_cmdline</span><span class="p">(</span><span class="s2">"down"</span><span class="p">)</span>
<span class="n">subprocess</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">scenario_config_file</span> <span class="o">=</span> <span class="n">app_config_file</span><span class="p">(</span><span class="n">config</span><span class="p">)</span>
<span class="n">os</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">scenario_config_file</span><span class="p">)</span>
<span class="n">scenario_docker_file</span> <span class="o">=</span> <span class="n">docker_compose_file</span><span class="p">(</span><span class="n">config</span><span class="p">)</span>
<span class="n">os</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">scenario_docker_file</span><span class="p">)</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">cli</span><span class="p">()</span>
</pre></div> </div> </div><p>where I added the commands <code>scenario up</code> <span class="callout">1</span> and <code>scenario down</code> <span class="callout">2</span>. As you can see the function <code>up</code> first copies the files <code>config/scenario.json</code> <span class="callout">3</span> and <code>docker/scenario.yml</code> <span class="callout">4</span> (that I still have to create) into files named after the scenario.</p><p>Then I run the command <code>up -d</code> <span class="callout">5</span> and wait for the database to be ready, as I already do for tests. After that, it's time to extract the port of the container with some very simple Python string processing <span class="callout">6</span> and to initialise the correct environment variable.</p><p>Last, I import and execute the Python file <span class="callout">7</span> containing the code of the scenario itself and print a friendly message with the command line to run <code>psql</code> <span class="callout">8</span> to have a Postgres shell into the newly created database.</p><p>The function <code>down</code> simply tears down the containers and removes the scenario configuration files.</p><p>The two missing config files are pretty simple. The docker compose configuration is</p><div class="code"><div class="title"><code>docker/scenario.yml</code></div><div class="content"><div class="highlight"><pre><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="s">'3.4'</span>
<span class="nt">services</span><span class="p">:</span>
<span class="w"> </span><span class="nt">db</span><span class="p">:</span>
<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgres</span>
<span class="w"> </span><span class="nt">environment</span><span class="p">:</span>
<span class="w"> </span><span class="nt">POSTGRES_DB</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_DB}</span>
<span class="w"> </span><span class="nt">POSTGRES_USER</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_USER}</span>
<span class="w"> </span><span class="nt">POSTGRES_PASSWORD</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_PASSWORD}</span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"5432"</span> <span class="callout">1</span>
<span class="w"> </span><span class="nt">web</span><span class="p">:</span>
<span class="w"> </span><span class="nt">build</span><span class="p">:</span>
<span class="w"> </span><span class="nt">context</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${PWD}</span>
<span class="w"> </span><span class="nt">dockerfile</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">docker/Dockerfile</span>
<span class="w"> </span><span class="nt">environment</span><span class="p">:</span>
<span class="w"> </span><span class="nt">FLASK_ENV</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${FLASK_ENV}</span>
<span class="w"> </span><span class="nt">FLASK_CONFIG</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${FLASK_CONFIG}</span>
<span class="w"> </span><span class="nt">APPLICATION_DB</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${APPLICATION_DB}</span>
<span class="w"> </span><span class="nt">POSTGRES_USER</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_USER}</span>
<span class="w"> </span><span class="nt">POSTGRES_HOSTNAME</span><span class="p">:</span><span class="w"> </span><span class="s">"db"</span>
<span class="w"> </span><span class="nt">POSTGRES_PASSWORD</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_PASSWORD}</span>
<span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">flask run --host 0.0.0.0</span>
<span class="w"> </span><span class="nt">volumes</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${PWD}:/opt/code</span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"5000"</span>
</pre></div> </div> </div><p>Here you can see that the database is ephemeral, that the port on the host is automatically assigned <span class="callout">1</span>, and that I also spin up the application (mapping it to a random port as well to avoid clashing with the development one).</p><p>The configuration file is</p><div class="code"><div class="title"><code>config/scenario.json</code></div><div class="content"><div class="highlight"><pre><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"FLASK_ENV"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"development"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"FLASK_CONFIG"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"development"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_DB"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"postgres"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_USER"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"postgres"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_HOSTNAME"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"localhost"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_PASSWORD"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"postgres"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"APPLICATION_DB"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application"</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">]</span>
</pre></div> </div> </div><p>which doesn't add anything new to what I already did for development and testing. </p><h3 id="git-commit-3262">Git commit</h3><p>You can see the changes made in this step through <a href="https://github.com/lgiordani/flask_project_setup/commit/dbb54d31af17866f9336199c65f1b495d879eb70">this Git commit</a> or <a href="https://github.com/lgiordani/flask_project_setup/tree/dbb54d31af17866f9336199c65f1b495d879eb70">browse the files</a>.</p><h3 id="resources-9e89">Resources</h3><ul><li><a href="https://docs.docker.com/compose/compose-file/#ports">Expose ports in docker-compose</a></li><li><a href="https://docs.docker.com/compose/reference/port/">Docker Compose port command</a> - A command to print the port exposed by a container</li><li><a href="https://www.postgresql.org/docs/current/app-psql.html">psql</a> - PostgreSQL interactive terminal</li></ul><h3 id="scenario-example-1-25a8">Scenario example 1</h3><p>Let's have a look at a very simple scenario that doesn't do anything on the database, just to understand the system. The code for the scenario is</p><div class="code"><div class="title"><code>scenarios/foo.py</code></div><div class="content"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">os</span>
<span class="k">def</span> <span class="nf">run</span><span class="p">():</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"HEY! This is scenario"</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"APPLICATION_SCENARIO_NAME"</span><span class="p">])</span>
</pre></div> </div> </div><p>When I run the scenario I get the following output</p><div class="code"><div class="content"><div class="highlight"><pre>$ ./manage.py scenario up foo
Creating network "scenario_foo_default" with the default driver
Creating scenario_foo_db_1 ... done
Creating scenario_foo_web_1 ... done
HEY! This is scenario foo
Your scenario is ready. If you want to open a SQL shell run
docker-compose -p scenario_foo -f docker/scenario_foo.yml exec db psql -U postgres -d application
</pre></div> </div> </div><p>The command <code>docker ps</code> shows that my development environment is happily running alongside with the scenario</p><div class="code"><div class="content"><div class="highlight"><pre>$ docker ps
CONTAINER ID IMAGE COMMAND [...] PORTS NAMES
85258892a2df scenario_foo_web "flask run --host 0.…" [...] 0.0.0.0:32826->5000/tcp scenario_foo_web_1
a031b6429e07 postgres "docker-entrypoint.s…" [...] 0.0.0.0:32827->5432/tcp scenario_foo_db_1
1a449d23da01 development_web "flask run --host 0.…" [...] 0.0.0.0:5000->5000/tcp development_web_1
28aa566321b5 postgres "docker-entrypoint.s…" [...] 0.0.0.0:5432->5432/tcp development_db_1
</pre></div> </div> </div><p>And the output of the command <code>scenario up foo</code> contains the string <code>HEY! This is scenario foo</code> that was printed by the file <code>foo.py</code>. We can also successfully run the suggested command</p><div class="code"><div class="content"><div class="highlight"><pre>$ docker-compose -p scenario_foo -f docker/scenario_foo.yml exec db psql -U postgres -d application
psql (12.3 (Debian 12.3-1.pgdg100+1))
Type "help" for help.
application=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-------------+----------+----------+------------+------------+-----------------------
application | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
application=#
</pre></div> </div> </div><p>And inside the database we find the database <code>application</code> created explicitly for the scenario (the name is specified in <code>config/scenario.json</code>). If you don't know <code>psql</code> you can exit with <code>\q</code> or <code>Ctrl-d</code>.</p><p>Before tearing down the scenario have a look at the two files <code>config/scenario_foo.json</code> and <code>docker/scenario_foo.yml</code>. They are just copies of <code>config/scenario.json</code> and <code>docker/scenario.yml</code> but I think seeing them there might help to understand how the whole thing works. When you are done run <code>./manage.py scenario down foo</code>.</p><h3 id="git-commit-3262">Git commit</h3><p>You can see the changes made in this step through <a href="https://github.com/lgiordani/flask_project_setup/commit/9d9601508cfa7dc5d718d76cd0827396069035fd">this Git commit</a> or <a href="https://github.com/lgiordani/flask_project_setup/tree/9d9601508cfa7dc5d718d76cd0827396069035fd">browse the files</a>.</p><h3 id="scenario-example-2-66ea">Scenario example 2</h3><p>Let's do something a bit more interesting. The new scenario is contained in <code>scenarios/users.py</code></p><div class="code"><div class="title"><code>scenarios/users.py</code></div><div class="content"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">application.app</span> <span class="kn">import</span> <span class="n">create_app</span>
<span class="kn">from</span> <span class="nn">application.models</span> <span class="kn">import</span> <span class="n">db</span><span class="p">,</span> <span class="n">User</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">create_app</span><span class="p">(</span><span class="s2">"development"</span><span class="p">)</span> <span class="callout">1</span>
<span class="k">def</span> <span class="nf">run</span><span class="p">():</span>
<span class="k">with</span> <span class="n">app</span><span class="o">.</span><span class="n">app_context</span><span class="p">():</span>
<span class="n">db</span><span class="o">.</span><span class="n">drop_all</span><span class="p">()</span>
<span class="n">db</span><span class="o">.</span><span class="n">create_all</span><span class="p">()</span>
<span class="c1"># Administrator</span>
<span class="n">admin</span> <span class="o">=</span> <span class="n">User</span><span class="p">(</span><span class="n">email</span><span class="o">=</span><span class="s2">"admin@server.com"</span><span class="p">)</span>
<span class="n">db</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">admin</span><span class="p">)</span> <span class="callout">2</span>
<span class="c1"># First user</span>
<span class="n">user1</span> <span class="o">=</span> <span class="n">User</span><span class="p">(</span><span class="n">email</span><span class="o">=</span><span class="s2">"user1@server.com"</span><span class="p">)</span>
<span class="n">db</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">user1</span><span class="p">)</span>
<span class="c1"># Second user</span>
<span class="n">user2</span> <span class="o">=</span> <span class="n">User</span><span class="p">(</span><span class="n">email</span><span class="o">=</span><span class="s2">"user2@server.com"</span><span class="p">)</span>
<span class="n">db</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">user2</span><span class="p">)</span>
<span class="n">db</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
</pre></div> </div> </div><p>I decided to be as agnostic as possible in the scenarios, to avoid creating something too specific that eventually would not give me enough flexibility to test what I need. This means that the scenario has to create the app <span class="callout">1</span> and to use the database session explicitly <span class="callout">2</span>, as I do in this example. The application is created with the configuration <code>"development"</code> <span class="callout">1</span>. Remember that this is the Flask configuration that you find in <code>application/config.py</code>, not the one that is in <code>config/development.json</code>.</p><p>I can run the scenario with</p><div class="code"><div class="content"><div class="highlight"><pre>$ ./manage.py scenario up users
</pre></div> </div> </div><p>and then connect to the database to find my users</p><div class="code"><div class="content"><div class="highlight"><pre>$ docker-compose -p scenario_users -f docker/scenario_users.yml exec db psql -U postgres -d application
psql (12.3 (Debian 12.3-1.pgdg100+1))
Type "help" for help.
application=# \dt
List of relations
Schema | Name | Type | Owner
--------+-------+-------+----------
public | users | table | postgres
(1 row)
application=# select * from users;
id | email
----+------------------
1 | admin@server.com
2 | user1@server.com
3 | user2@server.com
(3 rows)
application=# \q
</pre></div> </div> </div><h3 id="git-commit-3262">Git commit</h3><p>You can see the changes made in this step through <a href="https://github.com/lgiordani/flask_project_setup/commit/b475c5e3d455098691fa1c736de573182d0e44ec">this Git commit</a> or <a href="https://github.com/lgiordani/flask_project_setup/tree/b475c5e3d455098691fa1c736de573182d0e44ec">browse the files</a>.</p>
<div class="advertisement">
<a href="https://www.thedigitalcat.academy/freebie-first-class-objects">
<img src="/images/first-class-objects/cover.jpg" />
</a>
<div class="body">
<h2 id="first-class-objects-in-python-fffa">First-class objects in Python<a class="headerlink" href="#first-class-objects-in-python-fffa" title="Permanent link">¶</a></h2>
<p>Higher-order functions, wrappers, and factories</p>
<p>Learn all you need to know to understand first-class citizenship in Python, the gateway to grasp how decorators work and how functional programming can supercharge your code.</p>
<div class="actions">
<a class="action" href="https://www.thedigitalcat.academy/freebie-first-class-objects">Get your FREE copy</a>
</div>
</div>
</div>
<h2 id="step-2---simulating-the-production-environment-80ac">Step 2 - Simulating the production environment<a class="headerlink" href="#step-2---simulating-the-production-environment-80ac" title="Permanent link">¶</a></h2><p>As I stated at the very beginning of this mini series of posts, one of my goals was to run in development the same database that I run in production, and for this reason I went through the configuration steps that allowed me to have a Postgres container running both in development and during tests. In a real production scenario Postgres would probably run in a separate instance, for example on the RDS service in AWS, but as long as you have the connection parameters nothing changes in the configuration.</p><p>Docker actually allows us to easily simulate the production environment. If our notebook was connected 24/7 we might as well host the production there directly. Not that I recommend this nowadays, but this is how many important companies begun many years ago when cloud computing had not been here yet. Instead of installing a LAMP stack we configure containers, but the idea doesn't change.</p><p>I will then create a configuration that simulates a production environment and then give some hints on how to translate this into a proper production infrastructure. If you want to have a clear picture of the components of a web application in production read my post <a href="https://www.thedigitalcatonline.com/blog/2020/02/16/dissecting-a-web-stack/">Dissecting a web stack</a> that analyses them one by one.</p><p>The first component that we have to change here is the HTTP server. In development we use Flask's development server, and the first message that server prints is <code>WARNING: This is a development server. Do not use it in a production deployment.</code> Got it, Flask! A good choice to replace it is Gunicorn, so first of all I add it in the requirements</p><div class="code"><div class="title"><code>requirements/production.txt</code></div><div class="content"><div class="highlight"><pre>Flask
flask-sqlalchemy
psycopg2
flask-migrate
<span class="hll">gunicorn
</pre></div> </div> </div><p>Then I need to create a docker-compose configuration for production</p><div class="code"><div class="title"><code>docker/production.yml</code></div><div class="content"><div class="highlight"><pre><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="s">'3.4'</span>
<span class="nt">services</span><span class="p">:</span>
<span class="w"> </span><span class="nt">db</span><span class="p">:</span>
<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgres</span>
<span class="w"> </span><span class="nt">environment</span><span class="p">:</span>
<span class="w"> </span><span class="nt">POSTGRES_DB</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_DB}</span>
<span class="w"> </span><span class="nt">POSTGRES_USER</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_USER}</span>
<span class="w"> </span><span class="nt">POSTGRES_PASSWORD</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_PASSWORD}</span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"${POSTGRES_PORT}:5432"</span>
<span class="w"> </span><span class="nt">volumes</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">pgdata:/var/lib/postgresql/data</span>
<span class="w"> </span><span class="nt">web</span><span class="p">:</span>
<span class="w"> </span><span class="nt">build</span><span class="p">:</span>
<span class="w"> </span><span class="nt">context</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${PWD}</span>
<span class="w"> </span><span class="nt">dockerfile</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">docker/Dockerfile.production</span>
<span class="w"> </span><span class="nt">environment</span><span class="p">:</span>
<span class="w"> </span><span class="nt">FLASK_ENV</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${FLASK_ENV}</span>
<span class="w"> </span><span class="nt">FLASK_CONFIG</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${FLASK_CONFIG}</span>
<span class="w"> </span><span class="nt">APPLICATION_DB</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${APPLICATION_DB}</span>
<span class="w"> </span><span class="nt">POSTGRES_USER</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_USER}</span>
<span class="w"> </span><span class="nt">POSTGRES_HOSTNAME</span><span class="p">:</span><span class="w"> </span><span class="s">"db"</span>
<span class="w"> </span><span class="nt">POSTGRES_PASSWORD</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_PASSWORD}</span>
<span class="w"> </span><span class="nt">POSTGRES_PORT</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_PORT}</span>
<span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">gunicorn -w 4 -b 0.0.0.0 wsgi:app</span> <span class="callout">1</span>
<span class="w"> </span><span class="nt">volumes</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${PWD}:/opt/code</span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"8000:8000"</span> <span class="callout">2</span>
<span class="nt">volumes</span><span class="p">:</span>
<span class="w"> </span><span class="nt">pgdata</span><span class="p">:</span>
</pre></div> </div> </div><p>As you can see here the command that runs the application is slightly different <span class="callout">1</span>. It exposes 4 processes (<code>-w 4</code>) on the container's address 0.0.0.0 loading the object <code>app</code> from the file <code>wsgi.py</code> (<code>wsgi:app</code>). As by default Gunicorn exposes port 8000 I mapped that <span class="callout">2</span> to the same port in the host.</p><p>Then I created the file <code>Dockerfile.production</code> that defines the production image of the web application</p><div class="code"><div class="title"><code>docker/Dockerfile.production</code></div><div class="content"><div class="highlight"><pre><span class="k">FROM</span><span class="w"> </span><span class="s">python:3</span>
<span class="k">ENV</span><span class="w"> </span>PYTHONUNBUFFERED<span class="w"> </span><span class="m">1</span>
<span class="k">RUN</span><span class="w"> </span>mkdir<span class="w"> </span>/opt/code
<span class="k">RUN</span><span class="w"> </span>mkdir<span class="w"> </span>/opt/requirements
<span class="k">WORKDIR</span><span class="w"> </span><span class="s">/opt/code</span>
<span class="k">ADD</span><span class="w"> </span>requirements<span class="w"> </span>/opt/requirements
<span class="k">RUN</span><span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>-r<span class="w"> </span>/opt/requirements/production.txt
</pre></div> </div> </div><p>The last thing I need is a configuration file</p><div class="code"><div class="title"><code>config/production.json</code></div><div class="content"><div class="highlight"><pre><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"FLASK_ENV"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"production"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"FLASK_CONFIG"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"production"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_DB"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"postgres"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_USER"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"postgres"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_HOSTNAME"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"localhost"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_PORT"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"5432"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_PASSWORD"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"postgres"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"APPLICATION_DB"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application"</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">]</span>
</pre></div> </div> </div><p>as you can notice this is not very different from the development one, as I just changed the values of <code>FLASK_ENV</code> and <code>FLASK_CONFIG</code>. Clearly this contains a secret that shouldn't be written in plain text, <code>POSTGRES_PASSWORD</code>, but after all this is a simulation of production. In a real environment secrets should be kept in an encrypted manager such as <a href="https://aws.amazon.com/secrets-manager/">AWS Secrets Manager</a>.</p><p>Remember that <code>FLASK_ENV</code> changes the internal settings of Flask, most notably disabling the debugger, and that <code>FLASK_CONFIG=production</code> loads the object <code>ProductionConfig</code> from <code>application/config.py</code>. That object is empty for the moment, but it might contain public configuration for the production server.</p><p>I can now build the image with</p><div class="code"><div class="content"><div class="highlight"><pre>$ APPLICATION_CONFIG="production" ./manage.py compose build web
</pre></div> </div> </div><h3 id="git-commit-3262">Git commit</h3><p>You can see the changes made in this step through <a href="https://github.com/lgiordani/flask_project_setup/commit/1c0fcf13c54ea13bb6e1307452de00529dbd57af">this Git commit</a> or <a href="https://github.com/lgiordani/flask_project_setup/tree/1c0fcf13c54ea13bb6e1307452de00529dbd57af">browse the files</a>.</p><h3 id="resources-9e89">Resources</h3><ul><li><a href="https://gunicorn.org/">Gunicorn</a> - A Python WSGI HTTP Server</li></ul><h2 id="step-3---scale-up-ce9e">Step 3 - Scale up<a class="headerlink" href="#step-3---scale-up-ce9e" title="Permanent link">¶</a></h2><p>Mapping the container port to the host is not a great idea, though, as it makes it impossible to scale up and down to serve more load, which is the main point of running containers in production. This might be solved in many ways in the cloud, for example in AWS you might run the container in AWS Fargate and register them in an Application Load Balancer. Another way to do it on a single host is to run a Web Server in front of your HTTP server, and this might be easily implemented with Docker Compose.</p><p>I will add nginx and serve HTTP from there, reverse proxying the application containers through docker-compose networking. First of all the new configuration for docker-compose</p><div class="code"><div class="title"><code>docker/production.yml</code></div><div class="content"><div class="highlight"><pre><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="s">'3.4'</span>
<span class="nt">services</span><span class="p">:</span>
<span class="w"> </span><span class="nt">db</span><span class="p">:</span>
<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgres</span>
<span class="w"> </span><span class="nt">environment</span><span class="p">:</span>
<span class="w"> </span><span class="nt">POSTGRES_DB</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_DB}</span>
<span class="w"> </span><span class="nt">POSTGRES_USER</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_USER}</span>
<span class="w"> </span><span class="nt">POSTGRES_PASSWORD</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_PASSWORD}</span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"${POSTGRES_PORT}:5432"</span>
<span class="w"> </span><span class="nt">volumes</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">pgdata:/var/lib/postgresql/data</span>
<span class="w"> </span><span class="nt">web</span><span class="p">:</span>
<span class="w"> </span><span class="nt">build</span><span class="p">:</span>
<span class="w"> </span><span class="nt">context</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${PWD}</span>
<span class="w"> </span><span class="nt">dockerfile</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">docker/Dockerfile.production</span>
<span class="w"> </span><span class="nt">environment</span><span class="p">:</span>
<span class="w"> </span><span class="nt">FLASK_ENV</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${FLASK_ENV}</span>
<span class="w"> </span><span class="nt">FLASK_CONFIG</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${FLASK_CONFIG}</span>
<span class="w"> </span><span class="nt">APPLICATION_DB</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${APPLICATION_DB}</span>
<span class="w"> </span><span class="nt">POSTGRES_USER</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_USER}</span>
<span class="w"> </span><span class="nt">POSTGRES_HOSTNAME</span><span class="p">:</span><span class="w"> </span><span class="s">"db"</span>
<span class="w"> </span><span class="nt">POSTGRES_PASSWORD</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_PASSWORD}</span>
<span class="w"> </span><span class="nt">POSTGRES_PORT</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_PORT}</span>
<span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">gunicorn -w 4 -b 0.0.0.0 wsgi:app</span>
<span class="w"> </span><span class="nt">volumes</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${PWD}:/opt/code</span>
<span class="hll"><span class="w"> </span><span class="nt">nginx</span><span class="p">:</span>
</span><span class="hll"><span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">nginx</span>
</span><span class="hll"><span class="w"> </span><span class="nt">volumes</span><span class="p">:</span>
</span><span class="hll"><span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">./nginx/nginx.conf:/etc/nginx/nginx.conf:ro</span>
</span><span class="hll"><span class="w"> </span><span class="nt">ports</span><span class="p">:</span>
</span><span class="hll"><span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">8080:8080</span>
</span>
<span class="nt">volumes</span><span class="p">:</span>
<span class="w"> </span><span class="nt">pgdata</span><span class="p">:</span>
</pre></div> </div> </div><p>As you can see I added a service <code>nginx</code> that runs the default Nginx image, mapping a custom configuration file that I will create in a minute. The application container doesn't need any port mapping, as I won't access it directly from the host anymore. The Nginx configuration file is</p><div class="code"><div class="title"><code>docker/nginx/nginx.conf</code></div><div class="content"><div class="highlight"><pre><span class="k">worker_processes</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span>
<span class="k">events</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kn">worker_connections</span><span class="w"> </span><span class="mi">1024</span><span class="p">;</span><span class="w"> </span><span class="p">}</span>
<span class="k">http</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kn">sendfile</span><span class="w"> </span><span class="no">on</span><span class="p">;</span>
<span class="w"> </span><span class="kn">upstream</span><span class="w"> </span><span class="s">app</span><span class="w"> </span><span class="p">{</span> <span class="callout">1</span>
<span class="w"> </span><span class="kn">server</span><span class="w"> </span><span class="n">web</span><span class="p">:</span><span class="mi">8000</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="kn">server</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kn">listen</span><span class="w"> </span><span class="mi">8080</span><span class="p">;</span> <span class="callout">2</span>
<span class="w"> </span><span class="kn">location</span><span class="w"> </span><span class="s">/</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kn">proxy_pass</span><span class="w"> </span><span class="s">http://app</span><span class="p">;</span>
<span class="w"> </span><span class="kn">proxy_redirect</span><span class="w"> </span><span class="no">off</span><span class="p">;</span>
<span class="w"> </span><span class="kn">proxy_set_header</span><span class="w"> </span><span class="s">Host</span><span class="w"> </span><span class="nv">$host</span><span class="p">;</span>
<span class="w"> </span><span class="kn">proxy_set_header</span><span class="w"> </span><span class="s">X-Real-IP</span><span class="w"> </span><span class="nv">$remote_addr</span><span class="p">;</span>
<span class="w"> </span><span class="kn">proxy_set_header</span><span class="w"> </span><span class="s">X-Forwarded-For</span><span class="w"> </span><span class="nv">$proxy_add_x_forwarded_for</span><span class="p">;</span>
<span class="w"> </span><span class="kn">proxy_set_header</span><span class="w"> </span><span class="s">X-Forwarded-Host</span><span class="w"> </span><span class="nv">$server_name</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</pre></div> </div> </div><p>This is a pretty standard configuration, and in a real production environment I would add many other configuration values (most notably serving HTTPS instead of HTTP). The section <code>upstream</code> <span class="callout">1</span> leverages docker-compose networking referring to <code>web</code>, which in the internal DNS directly maps to the IPs of the service with the same name. The port 8000 comes from the default Gunicorn port that I already mentioned before. I won't run the nginx container as root on my notebook, so I expose port 8080 <span class="callout">2</span> instead of the traditional 80 for HTTP, and this is also something that might be different in a real production environment.</p><p>I can at this point run</p><div class="code"><div class="content"><div class="highlight"><pre>$ APPLICATION_CONFIG="production" ./manage.py compose up -d
Starting production_db_1 ... done
Starting production_nginx_1 ... done
Starting production_web_1 ... done
</pre></div> </div> </div><p>It's interesting to have a look at the logs of the nginx container, as Nginx by default prints all the incoming requests</p><div class="code"><div class="content"><div class="highlight"><pre>$ APPLICATION_CONFIG="production" ./manage.py compose logs -f nginx
Attaching to production_nginx_1
[...]
nginx_1 | 172.30.0.1 - - [05/Jul/2020:10:40:44 +0000] "GET / HTTP/1.1" 200 13 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"
</pre></div> </div> </div><p>The last line is what I get when I visit localhost:8080 while the production setup is up and running.</p><p>Scaling up and down the service is now a breeze</p><div class="code"><div class="content"><div class="highlight"><pre>$ APPLICATION_CONFIG="production" ./manage.py compose up -d --scale web=3
production_db_1 is up-to-date
Starting production_web_1 ...
Starting production_web_1 ... done
Creating production_web_2 ... done
Creating production_web_3 ... done
</pre></div> </div> </div><h3 id="git-commit-3262">Git commit</h3><p>You can see the changes made in this step through <a href="https://github.com/lgiordani/flask_project_setup/commit/2794ea1ca6e7c56a823ddf30201e69700f596bf2">this Git commit</a> or <a href="https://github.com/lgiordani/flask_project_setup/tree/2794ea1ca6e7c56a823ddf30201e69700f596bf2">browse the files</a>.</p><h3 id="resources-9e89">Resources</h3><ul><li><a href="https://nginx.org/en/">Nginx</a> - An HTTP and reverse proxy server (and more)</li><li><a href="https://hub.docker.com/_/nginx">Docker nginx</a> - the official nginx Docker image</li><li><a href="https://docs.docker.com/compose/reference/logs/">Docker Compose logs command</a> - A command to print container logs</li><li><a href="https://docs.docker.com/compose/reference/up/">Docker Compose up command</a> - The new way to scale containers in docker-compose</li></ul><h2 id="bonus-step---a-closer-look-at-docker-networking-f6d5">Bonus step - A closer look at Docker networking<a class="headerlink" href="#bonus-step---a-closer-look-at-docker-networking-f6d5" title="Permanent link">¶</a></h2><p>I mentioned that Docker Compose creates a connection between services, and used that in the configuration of the nginx container, but I understand that this might look like black magic to some people. While I believe that this is actually black magic, I also think that we can investigate it a bit, so let's open the grimoire and reveal (some of) the dark secrets of Docker networking.</p><p>While the production setup is running we can connect to the nginx container and see what is happening in real time, so first of all I run a bash shell on it</p><div class="code"><div class="content"><div class="highlight"><pre>$ APPLICATION_CONFIG="production" ./manage.py compose exec nginx bash
</pre></div> </div> </div><p>Once inside I can see my configuration file at <code>/etc/nginx/nginx.conf</code>, but this has not changed. Remember that Docker networking doesn't work as a templating engine, but with a local DNS. This means that if we try to resolve <code>web</code> from inside the container we should see multiple IPs. The command <code>dig</code> is a good tool to investigate the DNS, but it doesn't come preinstalled in the nginx container, so I need to run</p><div class="code"><div class="content"><div class="highlight"><pre>root@33cbaea369be:/# apt update && apt install dnsutils
</pre></div> </div> </div><p>and at this point I can run it</p><div class="code"><div class="content"><div class="highlight"><pre>root@33cbaea369be:/# dig web
; <<>> DiG 9.11.5-P4-5.1+deb10u1-Debian <<>> web
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30539
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;web. IN A
;; ANSWER SECTION:
web. 600 IN A 172.30.0.4
web. 600 IN A 172.30.0.6
web. 600 IN A 172.30.0.5
;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Sun Jul 05 10:58:18 UTC 2020
;; MSG SIZE rcvd: 78
root@33cbaea369be:/#
</pre></div> </div> </div><p>The command outputs 3 IPs, which correspond to the 3 containers of the service <code>web</code> that I am currently running. If I scale down (from outside the container)</p><div class="code"><div class="content"><div class="highlight"><pre>$ APPLICATION_CONFIG="production" ./manage.py compose up -d --scale web=1
</pre></div> </div> </div><p>then the output of <code>dig</code> becomes</p><div class="code"><div class="content"><div class="highlight"><pre>root@33cbaea369be:/# dig web
; <<>> DiG 9.11.5-P4-5.1+deb10u1-Debian <<>> web
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13146
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;web. IN A
;; ANSWER SECTION:
web. 600 IN A 172.30.0.4
;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Sun Jul 05 11:01:46 UTC 2020
;; MSG SIZE rcvd: 40
root@33cbaea369be:/#
</pre></div> </div> </div>
<div class="advertisement">
<a href="https://www.thedigitalcat.academy/freebie-first-class-objects">
<img src="/images/first-class-objects/cover.jpg" />
</a>
<div class="body">
<h2 id="first-class-objects-in-python-fffa">First-class objects in Python<a class="headerlink" href="#first-class-objects-in-python-fffa" title="Permanent link">¶</a></h2>
<p>Higher-order functions, wrappers, and factories</p>
<p>Learn all you need to know to understand first-class citizenship in Python, the gateway to grasp how decorators work and how functional programming can supercharge your code.</p>
<div class="actions">
<a class="action" href="https://www.thedigitalcat.academy/freebie-first-class-objects">Get your FREE copy</a>
</div>
</div>
</div>
<h2 id="how-to-create-the-production-infrastructure-d9bc">How to create the production infrastructure<a class="headerlink" href="#how-to-create-the-production-infrastructure-d9bc" title="Permanent link">¶</a></h2><p>This will be a very short section, as creating infrastructure and deploying in production are complex topics, so I want to just give some hints to stimulate your research.</p><p><a href="https://aws.amazon.com/ecs/">AWS ECS</a> is basically Docker in the cloud, and the whole structure can map almost 1 to 1 to the docker-compose setup, so it is worth learning. ECS can work on explicit EC2 instances that you manage, or in <a href="https://aws.amazon.com/fargate/">Fargate</a>, which means that the EC2 instances running the containers are transparently managed by AWS itself.</p><p><a href="https://www.terraform.io/">Terraform</a> is a good tool to create infrastructure. It has many limitations, mostly coming from its custom HCL language, but it's slowly becoming better (version 0.13 will finally allow us to run for loops on modules, for example). Despite its shortcomings, it's a great tool to create static infrastructure, so I recommend working on it.</p><p>Terraform is not the right tool to deploy your code, though, as that requires a dynamic interaction with the system, so you need to setup a good Continuous Integration system. <a href="https://www.jenkins.io/">Jenkins</a> is a very well known open source CI, but I personally ended up dropping it because it doesn't seem to be designed for large scale systems. For example, it is very complicated to automate the deploy of a Jenkins server, and dynamic large scale systems should require zero manual intervention to be created. Anyway, Jenkins is a good tool to start with, but you might want to have a look at other products like <a href="https://circleci.com/">CircleCI</a> or <a href="https://buildkite.com/">Buildkite</a>.</p><p>When you create your deploy pipeline you need to do much more than just creating the image and running it, at least for real applications. You need to decide when to apply database migrations and if you have a web front-end you will also need to compile and install the JavaScript assets. Since you don't want to have downtime when you deploy you will need to look into blue/green deployments, and in general to strategies that allow you to run different versions of the application at the same time, at least for short periods of time. Or for longer periods, if you want to perform A/B testing or zonal deployments.</p><h2 id="final-words-9803">Final words<a class="headerlink" href="#final-words-9803" title="Permanent link">¶</a></h2><p>This is the last post of this short series. I hope you learned something useful, and that it encouraged you to properly setup your projects and to investigate technologies like Docker. As always, feel free to send me feedback or questions, and if you find my posts useful please share them with whoever you thing might be interested.</p><h2 id="updates-0083">Updates<a class="headerlink" href="#updates-0083" title="Permanent link">¶</a></h2><p>2020-12-22 I reviewed the whole tutorial and corrected several typos</p><h2 id="feedback-d845">Feedback<a class="headerlink" href="#feedback-d845" title="Permanent link">¶</a></h2><p>Feel free to reach me on <a href="https://twitter.com/thedigicat">Twitter</a> if you have questions. The <a href="https://github.com/TheDigitalCatOnline/blog_source/issues">GitHub issues</a> page is the best place to submit corrections.</p>Flask project setup: TDD, Docker, Postgres and more - Part 22020-07-06T13:00:00+01:002021-02-23T20:00:00+00:00Leonardo Giordanitag:www.thedigitalcatonline.com,2020-07-06:/blog/2020/07/06/flask-project-setup-tdd-docker-postgres-and-more-part-2/<p>A step-by-step tutorial on how to setup a Flask project with TDD, Docker and Postgres</p><p>In this series of posts I explore the development of a Flask project with a setup that is built with efficiency and tidiness in mind, using TDD, Docker and Postgres.</p><h2 id="catch-up-7f97">Catch-up<a class="headerlink" href="#catch-up-7f97" title="Permanent link">¶</a></h2><p>In the <a href="https://www.thedigitalcatonline.com/blog/2020/07/05/flask-project-setup-tdd-docker-postgres-and-more-part-1/">previous post</a> we started from an empty project and learned how to add the minimal code to run a Flask project. Then we created a static configuration file and a management script that wraps the commands <code>flask</code> and <code>docker-compose</code> to run the application with a specific configuration.</p><p>In this post I will show you how to run a production-ready database alongside your code in a Docker container, both in your development setup and for the tests.</p><h2 id="step-1---adding-a-database-container-4920">Step 1 - Adding a database container<a class="headerlink" href="#step-1---adding-a-database-container-4920" title="Permanent link">¶</a></h2><p>A database is an integral part of a web application, so in this step I will add my database of choice, Postgres, to the project setup. To do this I need to add a service in the docker-compose configuration file</p><div class="code"><div class="title"><code>docker/development.yml</code></div><div class="content"><div class="highlight"><pre><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="s">'3.4'</span>
<span class="nt">services</span><span class="p">:</span>
<span class="w"> </span><span class="nt">db</span><span class="p">:</span>
<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgres</span>
<span class="w"> </span><span class="nt">environment</span><span class="p">:</span>
<span class="w"> </span><span class="nt">POSTGRES_DB</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_DB}</span> <span class="callout">1</span>
<span class="w"> </span><span class="nt">POSTGRES_USER</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_USER}</span>
<span class="w"> </span><span class="nt">POSTGRES_PASSWORD</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_PASSWORD}</span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"${POSTGRES_PORT}:5432"</span>
<span class="w"> </span><span class="nt">volumes</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">pgdata:/var/lib/postgresql/data</span> <span class="callout">2</span>
<span class="w"> </span><span class="nt">web</span><span class="p">:</span>
<span class="w"> </span><span class="nt">build</span><span class="p">:</span>
<span class="w"> </span><span class="nt">context</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${PWD}</span>
<span class="w"> </span><span class="nt">dockerfile</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">docker/Dockerfile</span>
<span class="w"> </span><span class="nt">environment</span><span class="p">:</span>
<span class="w"> </span><span class="nt">FLASK_ENV</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${FLASK_ENV}</span>
<span class="w"> </span><span class="nt">FLASK_CONFIG</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${FLASK_CONFIG}</span>
<span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">flask run --host 0.0.0.0</span>
<span class="w"> </span><span class="nt">volumes</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${PWD}:/opt/code</span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"5000:5000"</span>
<span class="nt">volumes</span><span class="p">:</span>
<span class="w"> </span><span class="nt">pgdata</span><span class="p">:</span> <span class="callout">3</span>
</pre></div> </div> </div><p>The variables starting with <code>POSTGRES_</code> are requested by the PostgreSQL Docker image. In particular, remember that <code>POSTGRESQL_DB</code> <span class="callout">1</span> is the database that gets created by default when you create the image, and also the one that contains data on other databases as well, so for the application we usually want to use a different one.</p><p>Notice also that I'm creating a persistent volume for the service <code>db</code> <span class="callout">2</span> <span class="callout">3</span>, so that the content of the database is not lost when we tear down the container. For this service I'm using the default image, so no build step is needed.</p><p>To orchestrate this setup we need to add those variables to the JSON configuration</p><div class="code"><div class="title"><code>config/development.json</code></div><div class="content"><div class="highlight"><pre><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"FLASK_ENV"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"development"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"FLASK_CONFIG"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"development"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_DB"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"postgres"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_USER"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"postgres"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_HOSTNAME"</span><span class="p">,</span> <span class="callout">1</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"localhost"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_PORT"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"5432"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_PASSWORD"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"postgres"</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">]</span>
</pre></div> </div> </div><p>These are all development variables so there are no secrets. In production we will need a way to keep the secrets in a safe place and convert them into environment variables. The <a href="https://aws.amazon.com/secrets-manager/">AWS Secrets Manager</a> for example can directly map secrets into environment variables passed to the containers, saving you from having to explicitly connect to the service with the API.</p><p>You may have noticed the variable <code>POSTGRES_HOSTNAME</code> <span class="callout">1</span>, which is not used in the Docker Compose file. Generally we want the database to be accessible by utility scripts, so we want to record the host name where the database is running. As we will see shortly, other Docker containers do not need this, but migrations will.</p><p>We can run the commands <code>./manage.py compose up -d</code> and <code>./manage.py compose down</code> here to check that the database container works properly. Please note that the first time you run the command <code>compose -d</code> Docker will create the volume and build the Postgres image, and this might take some time.</p><div class="code"><div class="content"><div class="highlight"><pre>CONTAINER ID IMAGE COMMAND ... PORTS NAMES
9b5828dccd1c docker_web "flask run --host 0.…" ... 0.0.0.0:5000->5000/tcp docker_web_1
4440a18a1527 postgres "docker-entrypoint.s…" ... 0.0.0.0:5432->5432/tcp docker_db_1
</pre></div> </div> </div><p>Now we need to connect the application to the database and to do this we can leverage flask-sqlalchemy. As we will use this at every stage of the life of the application, the requirement goes among the production ones. We also need psycopg2 as it is the library used to connect to Postgres.</p><div class="code"><div class="title"><code>requirements/production.txt</code></div><div class="content"><div class="highlight"><pre>Flask
<span class="hll">flask-sqlalchemy
</span><span class="hll">psycopg2
</pre></div> </div> </div><p>Remember to run <code>pip install -r requirements/development.txt</code> to install the requirements locally and <code>./manage.py compose build web</code> to rebuild the image.</p><p>At this point I need to create a connection string in the configuration of the application. The connection string parameters come from the same environment variables used to spin up the container <code>db</code></p><div class="code"><div class="title"><code>application/config.py</code></div><div class="content"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">os</span>
<span class="k">class</span> <span class="nc">Config</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Base configuration"""</span>
<span class="n">user</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"POSTGRES_USER"</span><span class="p">]</span>
<span class="n">password</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"POSTGRES_PASSWORD"</span><span class="p">]</span>
<span class="n">hostname</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"POSTGRES_HOSTNAME"</span><span class="p">]</span>
<span class="n">port</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"POSTGRES_PORT"</span><span class="p">]</span>
<span class="n">database</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"APPLICATION_DB"</span><span class="p">]</span> <span class="callout">1</span>
<span class="n">SQLALCHEMY_DATABASE_URI</span> <span class="o">=</span> <span class="p">(</span>
<span class="sa">f</span><span class="s2">"postgresql+psycopg2://</span><span class="si">{</span><span class="n">user</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">password</span><span class="si">}</span><span class="s2">@</span><span class="si">{</span><span class="n">hostname</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">port</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="n">database</span><span class="si">}</span><span class="s2">"</span>
<span class="p">)</span>
<span class="n">SQLALCHEMY_TRACK_MODIFICATIONS</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">class</span> <span class="nc">ProductionConfig</span><span class="p">(</span><span class="n">Config</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Production configuration"""</span>
<span class="k">class</span> <span class="nc">DevelopmentConfig</span><span class="p">(</span><span class="n">Config</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Development configuration"""</span>
<span class="k">class</span> <span class="nc">TestingConfig</span><span class="p">(</span><span class="n">Config</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Testing configuration"""</span>
<span class="n">TESTING</span> <span class="o">=</span> <span class="kc">True</span>
</pre></div> </div> </div><p>As you can see, here I use the variable <code>APPLICATION_DB</code> <span class="callout">1</span> and not <code>POSTGRES_DB</code>, so I need to specify that as well in the config file. The reason, as I mentioned before, is that we prefer to separate the default database, used by Postgres to manage all other databases, from the one used specifically by our application.</p><div class="code"><div class="title"><code>config/development.json</code></div><div class="content"><div class="highlight"><pre><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"FLASK_ENV"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"development"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"FLASK_CONFIG"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"development"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_DB"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"postgres"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_USER"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"postgres"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_HOSTNAME"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"localhost"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_PORT"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"5432"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_PASSWORD"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"postgres"</span>
<span class="w"> </span><span class="p">},</span>
<span class="hll"><span class="w"> </span><span class="p">{</span>
</span><span class="hll"><span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"APPLICATION_DB"</span><span class="p">,</span>
</span><span class="hll"><span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application"</span>
</span><span class="hll"><span class="w"> </span><span class="p">}</span>
</span><span class="p">]</span>
</pre></div> </div> </div><p>At this point the application container needs to access some of the Postgres environment variables and the <code>APPLICATION_DB</code> one</p><div class="code"><div class="title"><code>docker/development.yml</code></div><div class="content"><div class="highlight"><pre><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="s">'3.4'</span>
<span class="nt">services</span><span class="p">:</span>
<span class="w"> </span><span class="nt">db</span><span class="p">:</span>
<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgres</span>
<span class="w"> </span><span class="nt">environment</span><span class="p">:</span>
<span class="w"> </span><span class="nt">POSTGRES_DB</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_DB}</span>
<span class="w"> </span><span class="nt">POSTGRES_USER</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_USER}</span>
<span class="w"> </span><span class="nt">POSTGRES_PASSWORD</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_PASSWORD}</span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"${POSTGRES_PORT}:5432"</span>
<span class="w"> </span><span class="nt">volumes</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">pgdata:/var/lib/postgresql/data</span>
<span class="w"> </span><span class="nt">web</span><span class="p">:</span>
<span class="w"> </span><span class="nt">build</span><span class="p">:</span>
<span class="w"> </span><span class="nt">context</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${PWD}</span>
<span class="w"> </span><span class="nt">dockerfile</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">docker/Dockerfile</span>
<span class="w"> </span><span class="nt">environment</span><span class="p">:</span>
<span class="w"> </span><span class="nt">FLASK_ENV</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${FLASK_ENV}</span>
<span class="w"> </span><span class="nt">FLASK_CONFIG</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${FLASK_CONFIG}</span>
<span class="hll"><span class="w"> </span><span class="nt">APPLICATION_DB</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${APPLICATION_DB}</span>
</span><span class="hll"><span class="w"> </span><span class="nt">POSTGRES_USER</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_USER}</span>
</span><span class="hll"><span class="w"> </span><span class="nt">POSTGRES_HOSTNAME</span><span class="p">:</span><span class="w"> </span><span class="s">"db"</span>
</span><span class="hll"><span class="w"> </span><span class="nt">POSTGRES_PASSWORD</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_PASSWORD}</span>
</span><span class="hll"><span class="w"> </span><span class="nt">POSTGRES_PORT</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_PORT}</span>
</span><span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">flask run --host 0.0.0.0</span>
<span class="w"> </span><span class="nt">volumes</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${PWD}:/opt/code</span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"5000:5000"</span>
<span class="nt">volumes</span><span class="p">:</span>
<span class="w"> </span><span class="nt">pgdata</span><span class="p">:</span>
</pre></div> </div> </div><p>Please note that the container <code>web</code> receives the environment variables we pass to the Postgres container because it requires them to connect to the db. The variable <code>POSTGRES_HOSTNAME</code> is passed to give the application the address of the database, and thanks to Docker Compose internal DNS we can simply pass the name of the container. We could not pass the value <code>localhost</code>, as the application, which is running in a container, cannot access the host through that address (unless we use other network modes, which is not ideal).</p><p>Running compose now spins up both Flask and Postgres, but the application is not properly connected to the database yet.</p><p>Let's have a look inside the DB to see what our configuration created. First run <code>./manage.py compose up -d</code> to spin up the containers, then connect to the Postgres DB with <code>./manage.py compose exec db psql -U postgres</code>. Please note that we have to specify the user with <code>-U</code>. The default value is <code>root</code>, but we changed it to <code>postgres</code> with the variable <code>POSTGRES_USER</code>.</p><p>You should see a command line like</p><div class="code"><div class="content"><div class="highlight"><pre>$ ./manage.py compose exec db psql -U postgres
psql (13.0 (Debian 13.0-1.pgdg100+1))
Type "help" for help.
postgres=#
</pre></div> </div> </div><p>Also note that by default we are logging into the database called <code>postgres</code>, which was configured by the variable <code>POSTGRES_DB</code>. We can list the databases with <code>\l</code></p><div class="code"><div class="content"><div class="highlight"><pre>postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+------------+------------+-----------------------
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)
postgres=#
</pre></div> </div> </div><p>Last, note that the application database configured with <code>APPLICATION_DB</code> is not present because we haven't created it yet. All the environment variables prefixed by <code>POSTGRES_</code> are used automatically by the Docker image to perform the initial configuration, which is why the database <code>postgres</code> is already there.</p><p>You can exit <code>psql</code> with Ctrl-D or <code>exit</code>.</p><h3 id="git-commit-3262">Git commit</h3><p>You can see the changes made in this step through <a href="https://github.com/lgiordani/flask_project_setup/commit/e1af82cb302d669b1559b788c92aafeab1b427d5">this Git commit</a> or <a href="https://github.com/lgiordani/flask_project_setup/tree/e1af82cb302d669b1559b788c92aafeab1b427d5">browse the files</a>.</p><h3 id="resources-9e89">Resources</h3><ul><li><a href="https://hub.docker.com/_/postgres">Postgres Docker image</a></li><li><a href="https://flask-sqlalchemy.palletsprojects.com/en/2.x/">Flask-SQLAlchemy</a> - A Flask extension to work with SQLAlchemy</li><li><a href="https://docs.sqlalchemy.org/en/13/dialects/postgresql.html">SQLAlchemy and PostgreSQL</a> - The Python SQL Toolkit and ORM</li></ul>
<div class="advertisement">
<a href="https://www.thedigitalcat.academy/freebie-first-class-objects">
<img src="/images/first-class-objects/cover.jpg" />
</a>
<div class="body">
<h2 id="first-class-objects-in-python-fffa">First-class objects in Python<a class="headerlink" href="#first-class-objects-in-python-fffa" title="Permanent link">¶</a></h2>
<p>Higher-order functions, wrappers, and factories</p>
<p>Learn all you need to know to understand first-class citizenship in Python, the gateway to grasp how decorators work and how functional programming can supercharge your code.</p>
<div class="actions">
<a class="action" href="https://www.thedigitalcat.academy/freebie-first-class-objects">Get your FREE copy</a>
</div>
</div>
</div>
<h2 id="step-2---connecting-the-application-and-the-database-20bf">Step 2 - Connecting the application and the database<a class="headerlink" href="#step-2---connecting-the-application-and-the-database-20bf" title="Permanent link">¶</a></h2><p>To connect the Flask application with the database running in the container we need to initialise a <code>SQLAlchemy</code> object and add it to the application factory.</p><div class="code"><div class="title"><code>application/models.py</code></div><div class="content"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">flask_sqlalchemy</span> <span class="kn">import</span> <span class="n">SQLAlchemy</span>
<span class="n">db</span> <span class="o">=</span> <span class="n">SQLAlchemy</span><span class="p">()</span>
</pre></div> </div> </div><div class="code"><div class="title"><code>application/app.py</code></div><div class="content"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span>
<span class="k">def</span> <span class="nf">create_app</span><span class="p">(</span><span class="n">config_name</span><span class="p">):</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
<span class="n">config_module</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"application.config.</span><span class="si">{</span><span class="n">config_name</span><span class="o">.</span><span class="n">capitalize</span><span class="p">()</span><span class="si">}</span><span class="s2">Config"</span>
<span class="hll"> <span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">from_object</span><span class="p">(</span><span class="n">config_module</span><span class="p">)</span>
</span>
<span class="hll"> <span class="kn">from</span> <span class="nn">application.models</span> <span class="kn">import</span> <span class="n">db</span>
</span>
<span class="hll"> <span class="n">db</span><span class="o">.</span><span class="n">init_app</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
</span>
<span class="nd">@app</span><span class="o">.</span><span class="n">route</span><span class="p">(</span><span class="s2">"/"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">hello_world</span><span class="p">():</span>
<span class="k">return</span> <span class="s2">"Hello, World!"</span>
<span class="k">return</span> <span class="n">app</span>
</pre></div> </div> </div><p>A pretty standard way to manage the database in Flask is to use flask-migrate, that adds some commands that allow us to create migrations and apply them.</p><p>With flask-migrate you have to create the migrations folder once and for all with <code>flask db init</code> and then, every time you change your models, run <code>flask db migrate -m "Some message"</code> and <code>flask db upgrade</code>. As both <code>db init</code> and <code>db migrate</code> create files in the current directory we now face a problem that every Docker-based setup has to face: file permissions.</p><p>The situation is the following: the application is running in the Docker container as root, and there is no connection between the users namespace in the container and that of the host. The result is that if the Docker container creates files in a directory that is mounted from the host (like the one that contains the application code in our example), those files will result as belonging to <code>root</code>. While this doesn't make impossible to work (we usually can become <code>root</code> on our development machines), it is annoying to say the least. The solution is to run those commands from outside the container, but this requires the Flask application to be configured.</p><p>Fortunately I wrapped the command <code>flask</code> in the script <code>manage.py</code>, which loads all the required environment variables. Let's add flask-migrate to the production requirements</p><div class="code"><div class="title"><code>requirements/production.txt</code></div><div class="content"><div class="highlight"><pre>Flask
flask-sqlalchemy
psycopg2
<span class="hll">flask-migrate
</pre></div> </div> </div><p>Remember to run <code>pip install -r requirements/development.txt</code> to install the requirements locally and <code>./manage.py compose build web</code> to rebuild the image. Please note that you need the executable <code>pg_config</code> and some other development tools installed in your system. If you get an error message from <code>pip</code> please check the documentation of your operating system to find out what to do to install the required packages. For Ubuntu Linux I had to run <code>sudo apt install build-essential python3-dev libpq-dev</code>.</p><p>Now we can initialise a <code>Migrate</code> object and add it to the application factory</p><div class="code"><div class="title"><code>application/models.py</code></div><div class="content"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">flask_sqlalchemy</span> <span class="kn">import</span> <span class="n">SQLAlchemy</span>
<span class="hll"><span class="kn">from</span> <span class="nn">flask_migrate</span> <span class="kn">import</span> <span class="n">Migrate</span>
</span>
<span class="n">db</span> <span class="o">=</span> <span class="n">SQLAlchemy</span><span class="p">()</span>
<span class="hll"><span class="n">migrate</span> <span class="o">=</span> <span class="n">Migrate</span><span class="p">()</span>
</pre></div> </div> </div><div class="code"><div class="title"><code>application/app.py</code></div><div class="content"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span>
<span class="k">def</span> <span class="nf">create_app</span><span class="p">(</span><span class="n">config_name</span><span class="p">):</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
<span class="n">config_module</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"application.config.</span><span class="si">{</span><span class="n">config_name</span><span class="o">.</span><span class="n">capitalize</span><span class="p">()</span><span class="si">}</span><span class="s2">Config"</span>
<span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">from_object</span><span class="p">(</span><span class="n">config_module</span><span class="p">)</span>
<span class="hll"> <span class="kn">from</span> <span class="nn">application.models</span> <span class="kn">import</span> <span class="n">db</span><span class="p">,</span> <span class="n">migrate</span>
</span>
<span class="n">db</span><span class="o">.</span><span class="n">init_app</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
<span class="hll"> <span class="n">migrate</span><span class="o">.</span><span class="n">init_app</span><span class="p">(</span><span class="n">app</span><span class="p">,</span> <span class="n">db</span><span class="p">)</span>
</span>
<span class="nd">@app</span><span class="o">.</span><span class="n">route</span><span class="p">(</span><span class="s2">"/"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">hello_world</span><span class="p">():</span>
<span class="k">return</span> <span class="s2">"Hello, World!"</span>
<span class="k">return</span> <span class="n">app</span>
</pre></div> </div> </div><p>I can now run the database initialisation script</p><div class="code"><div class="content"><div class="highlight"><pre>$ ./manage.py flask db init
Creating directory /home/leo/devel/flask-tutorial/migrations ... done
Creating directory /home/leo/devel/flask-tutorial/migrations/versions ... done
Generating /home/leo/devel/flask-tutorial/migrations/env.py ... done
Generating /home/leo/devel/flask-tutorial/migrations/README ... done
Generating /home/leo/devel/flask-tutorial/migrations/script.py.mako ... done
Generating /home/leo/devel/flask-tutorial/migrations/alembic.ini ... done
Please edit configuration/connection/logging settings in '/home/leo/devel/flask-tutorial/migrations/alembic.ini' before proceeding.
</pre></div> </div> </div><p>And, when we will start creating models we will use the commands <code>./manage.py flask db migrate</code> and <code>./manage.py flask db upgrade</code>. You will find a complete example at the end of this post.</p><p>For the time being let's have a brief look at what was created here. The command <code>db init</code> created the directory <code>migrations</code> and inside it some default configuration files and templates. The migration scripts will be created in the directory <code>migrations/versions</code> but at the moment that directory is empty, as we have no models and we run no migrations (only the initialisation of the system). No changes have been made to the database. The command <code>db init</code> can be run even without running containers (you can remove the directory <code>migrations</code> and try it).</p><h3 id="git-commit-3262">Git commit</h3><p>You can see the changes made in this step through <a href="https://github.com/lgiordani/flask_project_setup/commit/ee7e2b32fb85b9a22b3e4ddb5de4e27e58884c25">this Git commit</a> or <a href="https://github.com/lgiordani/flask_project_setup/tree/ee7e2b32fb85b9a22b3e4ddb5de4e27e58884c25">browse the files</a>.</p><h3 id="resources-9e89">Resources</h3><ul><li><a href="https://flask-sqlalchemy.palletsprojects.com/en/2.x/">Flask-SQLAlchemy</a> - A Flask extension to work with SQLAlchemy</li><li><a href="https://flask-migrate.readthedocs.io/en/latest/">Flask-Migrate</a> - A Flask extension to handle database migrations with Alembic.</li></ul><h2 id="step-3---testing-setup-1cb6">Step 3 - Testing setup<a class="headerlink" href="#step-3---testing-setup-1cb6" title="Permanent link">¶</a></h2><p>I want to use a TDD approach as much as possible when developing my applications, so I need to setup a good testing environment upfront, and it has to be as ephemeral as possible. It is not unusual in big projects to create (or scale up) infrastructure components explicitly to run tests, and through Docker and docker-compose we can easily do the same. Namely, I will:</p><ol><li>Spin up a test database in a container without permanent volumes</li><li>Initialise it</li><li>Run all the tests against it</li><li>Tear down the container</li></ol><p>This approach has one big advantage, which is that it requires no previous setup and can this be executed on infrastructure created on the fly. It also has disadvantages, however, as it can slow down the testing part of the application, which should be as fast as possible in a TDD setup. Tests that involve the database, however, should be considered integration tests, and not run continuously in a TDD process, which is impossible (or very hard) when using a framework that merges the concept of entity and database model. If you want to know more about this read <a href="https://leanpub.com/clean-architectures-in-python">the book</a> that I wrote on the subject.</p><p>Another advantage of this setup is it that we might need other things during the test, e.g. Celery, other databases, other servers. They can all be created through the docker-compose file.</p><p>Generally speaking testing is an umbrella under which many different things can happen. As I will use pytest I can run the full suite, but I might want to select specific tests, mentioning a single file or using the powerful option <code>-k</code> that allows me to select tests by pattern-matching their name. For this reason I want to map the management command line to that of pytest.</p><p>Let's add pytest to the testing requirements, along with a couple of packages to monitor the test coverage</p><div class="code"><div class="title"><code>requirements/testing.txt</code></div><div class="content"><div class="highlight"><pre>-r production.txt
<span class="hll">pytest
</span><span class="hll">coverage
</span><span class="hll">pytest-cov
</pre></div> </div> </div><p>As you can see I also use the coverage plugin to keep an eye on how well I cover the code with the tests. Remember to run <code>pip install -r requirements/development.txt</code> to install the requirements locally and <code>./manage.py compose build web</code> to rebuild the image.</p><p>Warning: before you change the script <code>manage.py</code> make sure you terminate all the running containers running <code>./manage.py compose down</code>. The next version will change the naming convention for containers and you might end up with some stale containers and run into issues with the database.</p><div class="code"><div class="title"><code>manage.py</code></div><div class="content"><div class="highlight"><pre><span class="ch">#! /usr/bin/env python</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">signal</span>
<span class="kn">import</span> <span class="nn">subprocess</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="kn">import</span> <span class="nn">click</span>
<span class="c1"># Ensure an environment variable exists and has a value</span>
<span class="k">def</span> <span class="nf">setenv</span><span class="p">(</span><span class="n">variable</span><span class="p">,</span> <span class="n">default</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="n">variable</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="n">variable</span><span class="p">,</span> <span class="n">default</span><span class="p">)</span>
<span class="n">setenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">,</span> <span class="s2">"development"</span><span class="p">)</span> <span class="callout">2</span>
<span class="k">def</span> <span class="nf">configure_app</span><span class="p">(</span><span class="n">config</span><span class="p">):</span> <span class="callout">1</span>
<span class="c1"># Read configuration from the relative JSON file</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s2">"config"</span><span class="p">,</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">config</span><span class="si">}</span><span class="s2">.json"</span><span class="p">))</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">config_data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="c1"># Convert the config into a usable Python dictionary</span>
<span class="n">config_data</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">((</span><span class="n">i</span><span class="p">[</span><span class="s2">"name"</span><span class="p">],</span> <span class="n">i</span><span class="p">[</span><span class="s2">"value"</span><span class="p">])</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">config_data</span><span class="p">)</span>
<span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">config_data</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="n">setenv</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">group</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">cli</span><span class="p">():</span>
<span class="k">pass</span>
<span class="nd">@cli</span><span class="o">.</span><span class="n">command</span><span class="p">(</span><span class="n">context_settings</span><span class="o">=</span><span class="p">{</span><span class="s2">"ignore_unknown_options"</span><span class="p">:</span> <span class="kc">True</span><span class="p">})</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">argument</span><span class="p">(</span><span class="s2">"subcommand"</span><span class="p">,</span> <span class="n">nargs</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="n">click</span><span class="o">.</span><span class="n">Path</span><span class="p">())</span>
<span class="k">def</span> <span class="nf">flask</span><span class="p">(</span><span class="n">subcommand</span><span class="p">):</span>
<span class="n">configure_app</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">))</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"flask"</span><span class="p">]</span> <span class="o">+</span> <span class="nb">list</span><span class="p">(</span><span class="n">subcommand</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">KeyboardInterrupt</span><span class="p">:</span>
<span class="n">p</span><span class="o">.</span><span class="n">send_signal</span><span class="p">(</span><span class="n">signal</span><span class="o">.</span><span class="n">SIGINT</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">docker_compose_cmdline</span><span class="p">(</span><span class="n">config</span><span class="p">):</span> <span class="callout">3</span>
<span class="n">configure_app</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">))</span>
<span class="n">docker_compose_file</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s2">"docker"</span><span class="p">,</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">config</span><span class="si">}</span><span class="s2">.yml"</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isfile</span><span class="p">(</span><span class="n">docker_compose_file</span><span class="p">):</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">"The file </span><span class="si">{</span><span class="n">docker_compose_file</span><span class="si">}</span><span class="s2"> does not exist"</span><span class="p">)</span>
<span class="k">return</span> <span class="p">[</span>
<span class="s2">"docker-compose"</span><span class="p">,</span>
<span class="s2">"-p"</span><span class="p">,</span> <span class="callout">4</span>
<span class="n">config</span><span class="p">,</span>
<span class="s2">"-f"</span><span class="p">,</span>
<span class="n">docker_compose_file</span><span class="p">,</span>
<span class="p">]</span>
<span class="nd">@cli</span><span class="o">.</span><span class="n">command</span><span class="p">(</span><span class="n">context_settings</span><span class="o">=</span><span class="p">{</span><span class="s2">"ignore_unknown_options"</span><span class="p">:</span> <span class="kc">True</span><span class="p">})</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">argument</span><span class="p">(</span><span class="s2">"subcommand"</span><span class="p">,</span> <span class="n">nargs</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="n">click</span><span class="o">.</span><span class="n">Path</span><span class="p">())</span>
<span class="k">def</span> <span class="nf">compose</span><span class="p">(</span><span class="n">subcommand</span><span class="p">):</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="n">docker_compose_cmdline</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">))</span> <span class="o">+</span> <span class="nb">list</span><span class="p">(</span><span class="n">subcommand</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">KeyboardInterrupt</span><span class="p">:</span>
<span class="n">p</span><span class="o">.</span><span class="n">send_signal</span><span class="p">(</span><span class="n">signal</span><span class="o">.</span><span class="n">SIGINT</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="nd">@cli</span><span class="o">.</span><span class="n">command</span><span class="p">()</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">argument</span><span class="p">(</span><span class="s2">"filenames"</span><span class="p">,</span> <span class="n">nargs</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test</span><span class="p">(</span><span class="n">filenames</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"testing"</span> <span class="callout">5</span>
<span class="n">configure_app</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">))</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="n">docker_compose_cmdline</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">))</span> <span class="o">+</span> <span class="p">[</span><span class="s2">"up"</span><span class="p">,</span> <span class="s2">"-d"</span><span class="p">]</span>
<span class="n">subprocess</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="n">docker_compose_cmdline</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">))</span> <span class="o">+</span> <span class="p">[</span><span class="s2">"logs"</span><span class="p">,</span> <span class="s2">"db"</span><span class="p">]</span>
<span class="n">logs</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">check_output</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="k">while</span> <span class="s2">"ready to accept connections"</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">logs</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">"utf-8"</span><span class="p">):</span> <span class="callout">6</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span>
<span class="n">logs</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">check_output</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"pytest"</span><span class="p">,</span> <span class="s2">"-svv"</span><span class="p">,</span> <span class="s2">"--cov=application"</span><span class="p">,</span> <span class="s2">"--cov-report=term-missing"</span><span class="p">]</span>
<span class="n">cmdline</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">filenames</span><span class="p">)</span>
<span class="n">subprocess</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="n">docker_compose_cmdline</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">))</span> <span class="o">+</span> <span class="p">[</span><span class="s2">"down"</span><span class="p">]</span>
<span class="n">subprocess</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">cli</span><span class="p">()</span>
</pre></div> </div> </div><p>Notable changes are</p><ul><li>The environment configuration code is now in the function <code>configure_app</code> <span class="callout">1</span>. This allows me to force the variable <code>APPLICATION_CONFIG</code> inside the script <span class="callout">2</span> and then configure the environment, which saves me from having to call tests with <code>APPLICATION_CONFIG=testing flask test</code>.</li><li>Both commands <code>flask</code> and <code>compose</code> use the configuration <code>development</code>. Since that is the default value of the variable <code>APPLICATION_CONFIG</code> they just have to call the function <code>configure_app</code>.</li><li>The docker-compose command line is needed both in the commands <code>compose</code> and in <code>test</code>, so I isolated some code into a function called <code>docker_compose_cmdline</code> <span class="callout">3</span> which returns a list as needed by <code>subprocess</code> functions. The command line now uses also the option <code>-p</code> (project name) <span class="callout">4</span> to give a prefix to the containers. This way we can run tests while running the development server.</li><li>The command <code>test</code> forces <code>APPLICATION_CONFIG</code> to be <code>testing</code> <span class="callout">5</span>, which loads the file <code>config/testing.json</code>, then runs docker-compose using the file <code>docker/testing.yml</code> (both file have not been created yet), runs the pytest command line, and tears down the testing database container. Before running the tests the script waits for the service to be available <span class="callout">6</span>. Postgres doesn't allow connection until the database is ready to accept them.</li></ul><div class="code"><div class="title"><code>config/testing.json</code></div><div class="content"><div class="highlight"><pre><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"FLASK_ENV"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"production"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"FLASK_CONFIG"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"testing"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_DB"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"postgres"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_USER"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"postgres"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_HOSTNAME"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"localhost"</span> <span class="callout">2</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_PORT"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"5433"</span> <span class="callout">1</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"POSTGRES_PASSWORD"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"postgres"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"APPLICATION_DB"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"test"</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">]</span>
</pre></div> </div> </div><p>Note that here I specified the value <code>5433</code> for <code>POSTGRES_PORT</code> <span class="callout">1</span>. This allows us to spin up the test database container while the development one is running, as that will use port <code>5432</code> and you can't have two different containers using the same port on the host. A more general solution could be to leave Docker pick a random host port for the container and then use that, but this requires a bit more code to be properly implemented, so I will come back to this problem when setting up the scenarios.</p><p>Also note that I set the variable <code>POSTGRES_HOSTNAME</code> to <code>localhost</code> in this file <span class="callout">2</span>. We will run the tests on the local machine and not in a container, so we can't use the DNS provided by Docker Compose.</p><p>The last piece of setup that we need is the orchestration configuration for Docker Compose</p><div class="code"><div class="title"><code>docker/testing.yml</code></div><div class="content"><div class="highlight"><pre><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="s">'3.4'</span>
<span class="nt">services</span><span class="p">:</span>
<span class="w"> </span><span class="nt">db</span><span class="p">:</span>
<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgres</span>
<span class="w"> </span><span class="nt">environment</span><span class="p">:</span>
<span class="w"> </span><span class="nt">POSTGRES_DB</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_DB}</span>
<span class="w"> </span><span class="nt">POSTGRES_USER</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_USER}</span>
<span class="w"> </span><span class="nt">POSTGRES_PASSWORD</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${POSTGRES_PASSWORD}</span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"${POSTGRES_PORT}:5432"</span>
</pre></div> </div> </div><p>Now we can run <code>./manage.py test</code> and get</p><div class="code"><div class="content"><div class="highlight"><pre>Creating network "testing_default" with the default driver
Creating testing_db_1 ... done
========================== test session starts =========================
platform linux -- Python 3.7.5, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
-- /home/leo/devel/flask-tutorial/venv3/bin/python3
cachedir: .pytest_cache
rootdir: /home/leo/devel/flask-tutorial
plugins: cov-2.10.0
collected 0 items
Coverage.py warning: No data was collected. (no-data-collected)
----------- coverage: platform linux, python 3.7.5-final-0 -----------
Name Stmts Miss Cover Missing
-----------------------------------------------------
application/app.py 11 11 0% 1-21
application/config.py 13 13 0% 1-31
application/models.py 4 4 0% 1-5
-----------------------------------------------------
TOTAL 28 28 0%
======================== no tests ran in 0.07s =======================
Stopping testing_db_1 ... done
Removing testing_db_1 ... done
Removing network testing_default
</pre></div> </div> </div><p>Note that the command first creates the testing database container <code>testing_db_1</code>, then runs pytest, and finally stops and remove the container. This is exactly what we wanted to achieve to run tests in isolation. At the moment, however there are no tests, and the testing database is empty.</p><h3 id="git-commit-3262">Git commit</h3><p>You can see the changes made in this step through <a href="https://github.com/lgiordani/flask_project_setup/commit/1de9666f8d06e4061ba3513b1eadd869b6a8dd3d">this Git commit</a> or <a href="https://github.com/lgiordani/flask_project_setup/tree/1de9666f8d06e4061ba3513b1eadd869b6a8dd3d">browse the files</a>.</p><h3 id="resources-9e89">Resources</h3><ul><li><a href="https://docs.pytest.org/en/latest/">pytest</a> - A full-featured Python testing framework</li><li><a href="https://www.thedigitalcatonline.com/blog/2018/07/05/useful-pytest-command-line-options/">Useful pytest command line options</a></li></ul><h2 id="step-4---initialise-the-testing-database-5870">Step 4 - Initialise the testing database<a class="headerlink" href="#step-4---initialise-the-testing-database-5870" title="Permanent link">¶</a></h2><p>When you develop a web application and then run it in production, you typically create the database once and then upgrade it through migrations. When running tests we need to create the database every time, so I need to add a way to run SQL commands on the testing database before I run pytest.</p><p>As running sql commands directly on the the database is often useful I will create a function that wraps the boilerplate for the connection. The command that creates the initial database at that point will be trivial.</p><div class="code"><div class="title"><code>manage.py</code></div><div class="content"><div class="highlight"><pre><span class="ch">#! /usr/bin/env python</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">signal</span>
<span class="kn">import</span> <span class="nn">subprocess</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="kn">import</span> <span class="nn">click</span>
<span class="kn">import</span> <span class="nn">psycopg2</span>
<span class="kn">from</span> <span class="nn">psycopg2.extensions</span> <span class="kn">import</span> <span class="n">ISOLATION_LEVEL_AUTOCOMMIT</span>
<span class="c1"># Ensure an environment variable exists and has a value</span>
<span class="k">def</span> <span class="nf">setenv</span><span class="p">(</span><span class="n">variable</span><span class="p">,</span> <span class="n">default</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="n">variable</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="n">variable</span><span class="p">,</span> <span class="n">default</span><span class="p">)</span>
<span class="n">setenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">,</span> <span class="s2">"development"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">configure_app</span><span class="p">(</span><span class="n">config</span><span class="p">):</span>
<span class="c1"># Read configuration from the relative JSON file</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s2">"config"</span><span class="p">,</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">config</span><span class="si">}</span><span class="s2">.json"</span><span class="p">))</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">config_data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="c1"># Convert the config into a usable Python dictionary</span>
<span class="n">config_data</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">((</span><span class="n">i</span><span class="p">[</span><span class="s2">"name"</span><span class="p">],</span> <span class="n">i</span><span class="p">[</span><span class="s2">"value"</span><span class="p">])</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">config_data</span><span class="p">)</span>
<span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">config_data</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="n">setenv</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">group</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">cli</span><span class="p">():</span>
<span class="k">pass</span>
<span class="nd">@cli</span><span class="o">.</span><span class="n">command</span><span class="p">(</span><span class="n">context_settings</span><span class="o">=</span><span class="p">{</span><span class="s2">"ignore_unknown_options"</span><span class="p">:</span> <span class="kc">True</span><span class="p">})</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">argument</span><span class="p">(</span><span class="s2">"subcommand"</span><span class="p">,</span> <span class="n">nargs</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="n">click</span><span class="o">.</span><span class="n">Path</span><span class="p">())</span>
<span class="k">def</span> <span class="nf">flask</span><span class="p">(</span><span class="n">subcommand</span><span class="p">):</span>
<span class="n">configure_app</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">))</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"flask"</span><span class="p">]</span> <span class="o">+</span> <span class="nb">list</span><span class="p">(</span><span class="n">subcommand</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">KeyboardInterrupt</span><span class="p">:</span>
<span class="n">p</span><span class="o">.</span><span class="n">send_signal</span><span class="p">(</span><span class="n">signal</span><span class="o">.</span><span class="n">SIGINT</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">docker_compose_cmdline</span><span class="p">(</span><span class="n">config</span><span class="p">):</span>
<span class="n">configure_app</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">))</span>
<span class="n">docker_compose_file</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s2">"docker"</span><span class="p">,</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">config</span><span class="si">}</span><span class="s2">.yml"</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isfile</span><span class="p">(</span><span class="n">docker_compose_file</span><span class="p">):</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">"The file </span><span class="si">{</span><span class="n">docker_compose_file</span><span class="si">}</span><span class="s2"> does not exist"</span><span class="p">)</span>
<span class="k">return</span> <span class="p">[</span>
<span class="s2">"docker-compose"</span><span class="p">,</span>
<span class="s2">"-p"</span><span class="p">,</span>
<span class="n">config</span><span class="p">,</span>
<span class="s2">"-f"</span><span class="p">,</span>
<span class="n">docker_compose_file</span><span class="p">,</span>
<span class="p">]</span>
<span class="nd">@cli</span><span class="o">.</span><span class="n">command</span><span class="p">(</span><span class="n">context_settings</span><span class="o">=</span><span class="p">{</span><span class="s2">"ignore_unknown_options"</span><span class="p">:</span> <span class="kc">True</span><span class="p">})</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">argument</span><span class="p">(</span><span class="s2">"subcommand"</span><span class="p">,</span> <span class="n">nargs</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="n">click</span><span class="o">.</span><span class="n">Path</span><span class="p">())</span>
<span class="k">def</span> <span class="nf">compose</span><span class="p">(</span><span class="n">subcommand</span><span class="p">):</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="n">docker_compose_cmdline</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">))</span> <span class="o">+</span> <span class="nb">list</span><span class="p">(</span><span class="n">subcommand</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">KeyboardInterrupt</span><span class="p">:</span>
<span class="n">p</span><span class="o">.</span><span class="n">send_signal</span><span class="p">(</span><span class="n">signal</span><span class="o">.</span><span class="n">SIGINT</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">run_sql</span><span class="p">(</span><span class="n">statements</span><span class="p">):</span>
<span class="n">conn</span> <span class="o">=</span> <span class="n">psycopg2</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span>
<span class="n">dbname</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"POSTGRES_DB"</span><span class="p">),</span>
<span class="n">user</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"POSTGRES_USER"</span><span class="p">),</span>
<span class="n">password</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"POSTGRES_PASSWORD"</span><span class="p">),</span>
<span class="n">host</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"POSTGRES_HOSTNAME"</span><span class="p">),</span>
<span class="n">port</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"POSTGRES_PORT"</span><span class="p">),</span>
<span class="p">)</span>
<span class="n">conn</span><span class="o">.</span><span class="n">set_isolation_level</span><span class="p">(</span><span class="n">ISOLATION_LEVEL_AUTOCOMMIT</span><span class="p">)</span>
<span class="n">cursor</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
<span class="k">for</span> <span class="n">statement</span> <span class="ow">in</span> <span class="n">statements</span><span class="p">:</span>
<span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">statement</span><span class="p">)</span>
<span class="n">cursor</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="nd">@cli</span><span class="o">.</span><span class="n">command</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">create_initial_db</span><span class="p">():</span> <span class="callout">1</span>
<span class="n">configure_app</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">))</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">run_sql</span><span class="p">([</span><span class="sa">f</span><span class="s2">"CREATE DATABASE </span><span class="si">{</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'APPLICATION_DB'</span><span class="p">)</span><span class="si">}</span><span class="s2">"</span><span class="p">])</span>
<span class="k">except</span> <span class="n">psycopg2</span><span class="o">.</span><span class="n">errors</span><span class="o">.</span><span class="n">DuplicateDatabase</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">"The database </span><span class="si">{</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'APPLICATION_DB'</span><span class="p">)</span><span class="si">}</span><span class="s2"> already exists and will not be recreated"</span>
<span class="p">)</span>
<span class="nd">@cli</span><span class="o">.</span><span class="n">command</span><span class="p">()</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">argument</span><span class="p">(</span><span class="s2">"filenames"</span><span class="p">,</span> <span class="n">nargs</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test</span><span class="p">(</span><span class="n">filenames</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"testing"</span>
<span class="n">configure_app</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">))</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="n">docker_compose_cmdline</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">))</span> <span class="o">+</span> <span class="p">[</span><span class="s2">"up"</span><span class="p">,</span> <span class="s2">"-d"</span><span class="p">]</span>
<span class="n">subprocess</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="n">docker_compose_cmdline</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">))</span> <span class="o">+</span> <span class="p">[</span><span class="s2">"logs"</span><span class="p">,</span> <span class="s2">"db"</span><span class="p">]</span>
<span class="n">logs</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">check_output</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="k">while</span> <span class="s2">"ready to accept connections"</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">logs</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">"utf-8"</span><span class="p">):</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span>
<span class="n">logs</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">check_output</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">run_sql</span><span class="p">([</span><span class="sa">f</span><span class="s2">"CREATE DATABASE </span><span class="si">{</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'APPLICATION_DB'</span><span class="p">)</span><span class="si">}</span><span class="s2">"</span><span class="p">])</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"pytest"</span><span class="p">,</span> <span class="s2">"-svv"</span><span class="p">,</span> <span class="s2">"--cov=application"</span><span class="p">,</span> <span class="s2">"--cov-report=term-missing"</span><span class="p">]</span>
<span class="n">cmdline</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">filenames</span><span class="p">)</span>
<span class="n">subprocess</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="n">docker_compose_cmdline</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">))</span> <span class="o">+</span> <span class="p">[</span><span class="s2">"down"</span><span class="p">]</span>
<span class="n">subprocess</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">cli</span><span class="p">()</span>
</pre></div> </div> </div><p>As you can see I took the opportunity to write the command <code>create_initial_db</code> <span class="callout">1</span> as well, that just runs the very same SQL command that creates the testing database, but in any configuration I will use. </p><p>Before moving on I think it's time to refactor the file <code>manage.py</code>. Refactoring is not mandatory, but I feel like some parts of the script are not generic enough, and when I will add the scenarios I will definitely need my functions to be flexible.</p><p>The new script is</p><div class="code"><div class="title"><code>manage.py</code></div><div class="content"><div class="highlight"><pre><span class="ch">#! /usr/bin/env python</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">signal</span>
<span class="kn">import</span> <span class="nn">subprocess</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="kn">import</span> <span class="nn">click</span>
<span class="kn">import</span> <span class="nn">psycopg2</span>
<span class="kn">from</span> <span class="nn">psycopg2.extensions</span> <span class="kn">import</span> <span class="n">ISOLATION_LEVEL_AUTOCOMMIT</span>
<span class="c1"># Ensure an environment variable exists and has a value</span>
<span class="k">def</span> <span class="nf">setenv</span><span class="p">(</span><span class="n">variable</span><span class="p">,</span> <span class="n">default</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="n">variable</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="n">variable</span><span class="p">,</span> <span class="n">default</span><span class="p">)</span>
<span class="n">setenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">,</span> <span class="s2">"development"</span><span class="p">)</span>
<span class="n">APPLICATION_CONFIG_PATH</span> <span class="o">=</span> <span class="s2">"config"</span>
<span class="n">DOCKER_PATH</span> <span class="o">=</span> <span class="s2">"docker"</span>
<span class="k">def</span> <span class="nf">app_config_file</span><span class="p">(</span><span class="n">config</span><span class="p">):</span> <span class="callout">1</span>
<span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">APPLICATION_CONFIG_PATH</span><span class="p">,</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">config</span><span class="si">}</span><span class="s2">.json"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">docker_compose_file</span><span class="p">(</span><span class="n">config</span><span class="p">):</span> <span class="callout">2</span>
<span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">DOCKER_PATH</span><span class="p">,</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">config</span><span class="si">}</span><span class="s2">.yml"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">configure_app</span><span class="p">(</span><span class="n">config</span><span class="p">):</span>
<span class="c1"># Read configuration from the relative JSON file</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">app_config_file</span><span class="p">(</span><span class="n">config</span><span class="p">))</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">config_data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="c1"># Convert the config into a usable Python dictionary</span>
<span class="n">config_data</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">((</span><span class="n">i</span><span class="p">[</span><span class="s2">"name"</span><span class="p">],</span> <span class="n">i</span><span class="p">[</span><span class="s2">"value"</span><span class="p">])</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">config_data</span><span class="p">)</span>
<span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">config_data</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="n">setenv</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">group</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">cli</span><span class="p">():</span>
<span class="k">pass</span>
<span class="nd">@cli</span><span class="o">.</span><span class="n">command</span><span class="p">(</span><span class="n">context_settings</span><span class="o">=</span><span class="p">{</span><span class="s2">"ignore_unknown_options"</span><span class="p">:</span> <span class="kc">True</span><span class="p">})</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">argument</span><span class="p">(</span><span class="s2">"subcommand"</span><span class="p">,</span> <span class="n">nargs</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="n">click</span><span class="o">.</span><span class="n">Path</span><span class="p">())</span>
<span class="k">def</span> <span class="nf">flask</span><span class="p">(</span><span class="n">subcommand</span><span class="p">):</span>
<span class="n">configure_app</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">))</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"flask"</span><span class="p">]</span> <span class="o">+</span> <span class="nb">list</span><span class="p">(</span><span class="n">subcommand</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">KeyboardInterrupt</span><span class="p">:</span>
<span class="n">p</span><span class="o">.</span><span class="n">send_signal</span><span class="p">(</span><span class="n">signal</span><span class="o">.</span><span class="n">SIGINT</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">docker_compose_cmdline</span><span class="p">(</span><span class="n">commands_string</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span> <span class="callout">4</span>
<span class="n">config</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">)</span>
<span class="n">configure_app</span><span class="p">(</span><span class="n">config</span><span class="p">)</span>
<span class="n">compose_file</span> <span class="o">=</span> <span class="n">docker_compose_file</span><span class="p">(</span><span class="n">config</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isfile</span><span class="p">(</span><span class="n">compose_file</span><span class="p">):</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">"The file </span><span class="si">{</span><span class="n">compose_file</span><span class="si">}</span><span class="s2"> does not exist"</span><span class="p">)</span>
<span class="n">command_line</span> <span class="o">=</span> <span class="p">[</span>
<span class="s2">"docker-compose"</span><span class="p">,</span>
<span class="s2">"-p"</span><span class="p">,</span>
<span class="n">config</span><span class="p">,</span>
<span class="s2">"-f"</span><span class="p">,</span>
<span class="n">compose_file</span><span class="p">,</span>
<span class="p">]</span>
<span class="k">if</span> <span class="n">commands_string</span><span class="p">:</span>
<span class="n">command_line</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">commands_string</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">" "</span><span class="p">))</span>
<span class="k">return</span> <span class="n">command_line</span>
<span class="nd">@cli</span><span class="o">.</span><span class="n">command</span><span class="p">(</span><span class="n">context_settings</span><span class="o">=</span><span class="p">{</span><span class="s2">"ignore_unknown_options"</span><span class="p">:</span> <span class="kc">True</span><span class="p">})</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">argument</span><span class="p">(</span><span class="s2">"subcommand"</span><span class="p">,</span> <span class="n">nargs</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="n">click</span><span class="o">.</span><span class="n">Path</span><span class="p">())</span>
<span class="k">def</span> <span class="nf">compose</span><span class="p">(</span><span class="n">subcommand</span><span class="p">):</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="n">docker_compose_cmdline</span><span class="p">()</span> <span class="o">+</span> <span class="nb">list</span><span class="p">(</span><span class="n">subcommand</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">KeyboardInterrupt</span><span class="p">:</span>
<span class="n">p</span><span class="o">.</span><span class="n">send_signal</span><span class="p">(</span><span class="n">signal</span><span class="o">.</span><span class="n">SIGINT</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">run_sql</span><span class="p">(</span><span class="n">statements</span><span class="p">):</span>
<span class="n">conn</span> <span class="o">=</span> <span class="n">psycopg2</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span>
<span class="n">dbname</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"POSTGRES_DB"</span><span class="p">),</span>
<span class="n">user</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"POSTGRES_USER"</span><span class="p">),</span>
<span class="n">password</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"POSTGRES_PASSWORD"</span><span class="p">),</span>
<span class="n">host</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"POSTGRES_HOSTNAME"</span><span class="p">),</span>
<span class="n">port</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"POSTGRES_PORT"</span><span class="p">),</span>
<span class="p">)</span>
<span class="n">conn</span><span class="o">.</span><span class="n">set_isolation_level</span><span class="p">(</span><span class="n">ISOLATION_LEVEL_AUTOCOMMIT</span><span class="p">)</span>
<span class="n">cursor</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
<span class="k">for</span> <span class="n">statement</span> <span class="ow">in</span> <span class="n">statements</span><span class="p">:</span>
<span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">statement</span><span class="p">)</span>
<span class="n">cursor</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">wait_for_logs</span><span class="p">(</span><span class="n">cmdline</span><span class="p">,</span> <span class="n">message</span><span class="p">):</span> <span class="callout">3</span>
<span class="n">logs</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">check_output</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="k">while</span> <span class="n">message</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">logs</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">"utf-8"</span><span class="p">):</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span>
<span class="n">logs</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">check_output</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="nd">@cli</span><span class="o">.</span><span class="n">command</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">create_initial_db</span><span class="p">():</span>
<span class="n">configure_app</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">))</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">run_sql</span><span class="p">([</span><span class="sa">f</span><span class="s2">"CREATE DATABASE </span><span class="si">{</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'APPLICATION_DB'</span><span class="p">)</span><span class="si">}</span><span class="s2">"</span><span class="p">])</span>
<span class="k">except</span> <span class="n">psycopg2</span><span class="o">.</span><span class="n">errors</span><span class="o">.</span><span class="n">DuplicateDatabase</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">"The database </span><span class="si">{</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'APPLICATION_DB'</span><span class="p">)</span><span class="si">}</span><span class="s2"> already exists and will not be recreated"</span>
<span class="p">)</span>
<span class="nd">@cli</span><span class="o">.</span><span class="n">command</span><span class="p">()</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">argument</span><span class="p">(</span><span class="s2">"filenames"</span><span class="p">,</span> <span class="n">nargs</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test</span><span class="p">(</span><span class="n">filenames</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"testing"</span>
<span class="n">configure_app</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">))</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="n">docker_compose_cmdline</span><span class="p">(</span><span class="s2">"up -d"</span><span class="p">)</span>
<span class="n">subprocess</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="n">docker_compose_cmdline</span><span class="p">(</span><span class="s2">"logs db"</span><span class="p">)</span>
<span class="n">wait_for_logs</span><span class="p">(</span><span class="n">cmdline</span><span class="p">,</span> <span class="s2">"ready to accept connections"</span><span class="p">)</span>
<span class="n">run_sql</span><span class="p">([</span><span class="sa">f</span><span class="s2">"CREATE DATABASE </span><span class="si">{</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'APPLICATION_DB'</span><span class="p">)</span><span class="si">}</span><span class="s2">"</span><span class="p">])</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"pytest"</span><span class="p">,</span> <span class="s2">"-svv"</span><span class="p">,</span> <span class="s2">"--cov=application"</span><span class="p">,</span> <span class="s2">"--cov-report=term-missing"</span><span class="p">]</span>
<span class="n">cmdline</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">filenames</span><span class="p">)</span>
<span class="n">subprocess</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="n">docker_compose_cmdline</span><span class="p">(</span><span class="s2">"down"</span><span class="p">)</span>
<span class="n">subprocess</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">cli</span><span class="p">()</span>
</pre></div> </div> </div><p>Notable changes:</p><ul><li>I created two new functions <code>app_config_file</code> <span class="callout">1</span> and <code>docker_compose_file</code> <span class="callout">2</span> that encapsulate the creation of the file paths.</li><li>I isolated the code that waits for a message in the database container logs, creating the function <code>wait_for_logs</code> <span class="callout">3</span>.</li><li>The command <code>docker_compose_cmdline</code> <span class="callout">4</span> now receives a string and converts it into a list internally. This way expressing commands is more natural, as it doesn't require the ugly list syntax that <code>subprocess</code> works with.</li></ul><h3 id="git-commit-3262">Git commit</h3><p>You can see the changes made in this step through <a href="https://github.com/lgiordani/flask_project_setup/commit/1781b43a872e9518ae179e30e9e640a059d87cf6">this Git commit</a> or <a href="https://github.com/lgiordani/flask_project_setup/tree/1781b43a872e9518ae179e30e9e640a059d87cf6">browse the files</a>.</p><h3 id="resources-9e89">Resources</h3><ul><li><a href="https://www.psycopg.org/docs/">Psycopg</a> – PostgreSQL database adapter for Python</li></ul><h2 id="step-5---fixtures-for-tests-50a8">Step 5 - Fixtures for tests<a class="headerlink" href="#step-5---fixtures-for-tests-50a8" title="Permanent link">¶</a></h2><p>Pytest uses fixtures for tests, so we should prepare some basic ones that will be generally useful. First let's include <code>pytest-flask</code>, which provides already some basic fixtures</p><div class="code"><div class="title"><code>requirements/testing.txt</code></div><div class="content"><div class="highlight"><pre>-r production.txt
pytest
coverage
pytest-cov
<span class="hll">pytest-flask
</pre></div> </div> </div><p>Then add the fixtures <code>app</code> and <code>database</code> to the file <code>tests/conftest.py</code>. The first is required by pytest-flask itself (it's used by other fixtures) and the second one is useful every time you need to interact with the database itself.</p><div class="code"><div class="title"><code>tests/conftest.py</code></div><div class="content"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">pytest</span>
<span class="kn">from</span> <span class="nn">application.app</span> <span class="kn">import</span> <span class="n">create_app</span>
<span class="kn">from</span> <span class="nn">application.models</span> <span class="kn">import</span> <span class="n">db</span>
<span class="nd">@pytest</span><span class="o">.</span><span class="n">fixture</span>
<span class="k">def</span> <span class="nf">app</span><span class="p">():</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">create_app</span><span class="p">(</span><span class="s2">"testing"</span><span class="p">)</span>
<span class="k">return</span> <span class="n">app</span>
<span class="nd">@pytest</span><span class="o">.</span><span class="n">fixture</span><span class="p">(</span><span class="n">scope</span><span class="o">=</span><span class="s2">"function"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">database</span><span class="p">(</span><span class="n">app</span><span class="p">):</span>
<span class="k">with</span> <span class="n">app</span><span class="o">.</span><span class="n">app_context</span><span class="p">():</span>
<span class="n">db</span><span class="o">.</span><span class="n">drop_all</span><span class="p">()</span>
<span class="n">db</span><span class="o">.</span><span class="n">create_all</span><span class="p">()</span>
<span class="k">yield</span> <span class="n">db</span>
</pre></div> </div> </div><p>Remember to create the empty file <code>tests/__init__.py</code> to make pytest correctly load the code.</p><p>As you can see, the fixture <code>database</code> uses the methods <code>drop_all</code> and <code>create_all</code> to reset the database. The reason is that this fixture is recreated for each function, and we can't be sure a previous function left the database clean. As a matter of fact, we might be almost sure of the opposite.</p><h3 id="git-commit-3262">Git commit</h3><p>You can see the changes made in this step through <a href="https://github.com/lgiordani/flask_project_setup/commit/6f30e869ae040420c3b3b94d414b5dd841217c0d">this Git commit</a> or <a href="https://github.com/lgiordani/flask_project_setup/tree/6f30e869ae040420c3b3b94d414b5dd841217c0d">browse the files</a>.</p><h3 id="resources-9e89">Resources</h3><ul><li><a href="https://docs.pytest.org/en/stable/fixture.html">pytest fixtures</a> - One of the most powerful features of pytest</li><li><a href="https://pytest-flask.readthedocs.io/en/latest/">pytest-flask</a> - A plugin for pytest that simplifies testing Flask applications</li><li><a href="https://flask-sqlalchemy.palletsprojects.com/en/2.x/api/">Flask-SQLAlchemy API</a></li></ul><h2 id="bonus-step---a-full-tdd-example-19ec">Bonus step - A full TDD example<a class="headerlink" href="#bonus-step---a-full-tdd-example-19ec" title="Permanent link">¶</a></h2><p>Before wrapping up this post, I want to give you a full example of the TDD process that I would follow given the current state of the setup, which is already complete enough to start the development of an application. Let's pretend my goal is that of adding a <code>User</code> model that can be created with an <code>id</code> (primary key) and an <code>email</code> fields.</p><p>First of all I write a test that creates a user in the database and then retrieves it, checking its attributes</p><div class="code"><div class="title"><code>tests/test_user.py</code></div><div class="content"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">application.models</span> <span class="kn">import</span> <span class="n">User</span>
<span class="k">def</span> <span class="nf">test__create_user</span><span class="p">(</span><span class="n">database</span><span class="p">):</span>
<span class="n">email</span> <span class="o">=</span> <span class="s2">"some.email@server.com"</span>
<span class="n">user</span> <span class="o">=</span> <span class="n">User</span><span class="p">(</span><span class="n">email</span><span class="o">=</span><span class="n">email</span><span class="p">)</span>
<span class="n">database</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">user</span><span class="p">)</span>
<span class="n">database</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
<span class="n">user</span> <span class="o">=</span> <span class="n">User</span><span class="o">.</span><span class="n">query</span><span class="o">.</span><span class="n">first</span><span class="p">()</span>
<span class="k">assert</span> <span class="n">user</span><span class="o">.</span><span class="n">email</span> <span class="o">==</span> <span class="n">email</span>
</pre></div> </div> </div><p>Running this test results in an error, because the module <code>User</code> does not exist</p><div class="code"><div class="content"><div class="highlight"><pre>$<span class="w"> </span>./manage.py<span class="w"> </span><span class="nb">test</span>
Creating<span class="w"> </span>network<span class="w"> </span><span class="s2">"testing_default"</span><span class="w"> </span>with<span class="w"> </span>the<span class="w"> </span>default<span class="w"> </span>driver
Creating<span class="w"> </span>testing_db_1<span class="w"> </span>...<span class="w"> </span><span class="k">done</span>
<span class="o">=======================================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">======================================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.7.5,<span class="w"> </span>pytest-5.4.3,<span class="w"> </span>py-1.9.0,<span class="w"> </span>pluggy-0.13.1<span class="w"> </span>--
/home/leo/devel/flask-tutorial/venv3/bin/python3
cachedir:<span class="w"> </span>.pytest_cache
rootdir:<span class="w"> </span>/home/leo/devel/flask-tutorial
plugins:<span class="w"> </span>flask-1.0.0,<span class="w"> </span>cov-2.10.0
collected<span class="w"> </span><span class="m">0</span><span class="w"> </span>items<span class="w"> </span>/<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="nv">error</span>
<span class="o">=============================================</span><span class="w"> </span><span class="nv">ERRORS</span><span class="w"> </span><span class="o">=============================================</span>
___________________________<span class="w"> </span>ERROR<span class="w"> </span>collecting<span class="w"> </span>tests/tests/test_user.py<span class="w"> </span>___________________________
ImportError<span class="w"> </span><span class="k">while</span><span class="w"> </span>importing<span class="w"> </span><span class="nb">test</span><span class="w"> </span>module<span class="w"> </span><span class="s1">'/home/leo/devel/flask-tutorial/tests/tests/test_user.py'</span>.
Hint:<span class="w"> </span>make<span class="w"> </span>sure<span class="w"> </span>your<span class="w"> </span><span class="nb">test</span><span class="w"> </span>modules/packages<span class="w"> </span>have<span class="w"> </span>valid<span class="w"> </span>Python<span class="w"> </span>names.
Traceback:
venv3/lib/python3.7/site-packages/_pytest/python.py:511:<span class="w"> </span><span class="k">in</span><span class="w"> </span>_importtestmodule
<span class="w"> </span><span class="nv">mod</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>self.fspath.pyimport<span class="o">(</span><span class="nv">ensuresyspath</span><span class="o">=</span>importmode<span class="o">)</span>
venv3/lib/python3.7/site-packages/py/_path/local.py:704:<span class="w"> </span><span class="k">in</span><span class="w"> </span>pyimport
<span class="w"> </span>__import__<span class="o">(</span>modname<span class="o">)</span>
venv3/lib/python3.7/site-packages/_pytest/assertion/rewrite.py:152:<span class="w"> </span><span class="k">in</span><span class="w"> </span>exec_module
<span class="w"> </span>exec<span class="o">(</span>co,<span class="w"> </span>module.__dict__<span class="o">)</span>
tests/tests/test_user.py:1:<span class="w"> </span><span class="k">in</span><span class="w"> </span><module>
<span class="w"> </span>from<span class="w"> </span>application.models<span class="w"> </span>import<span class="w"> </span>User
<span class="hll">E<span class="w"> </span>ImportError:<span class="w"> </span>cannot<span class="w"> </span>import<span class="w"> </span>name<span class="w"> </span><span class="s1">'User'</span><span class="w"> </span>from<span class="w"> </span><span class="s1">'application.models'</span>
</span><span class="w"> </span><span class="o">(</span>/home/leo/devel/flask-tutorial/application/models.py<span class="o">)</span>
-----------<span class="w"> </span>coverage:<span class="w"> </span>platform<span class="w"> </span>linux,<span class="w"> </span>python<span class="w"> </span><span class="m">3</span>.7.5-final-0<span class="w"> </span>-----------
Name<span class="w"> </span>Stmts<span class="w"> </span>Miss<span class="w"> </span>Cover<span class="w"> </span>Missing
-----------------------------------------------------
application/app.py<span class="w"> </span><span class="m">11</span><span class="w"> </span><span class="m">9</span><span class="w"> </span><span class="m">18</span>%<span class="w"> </span><span class="m">6</span>-21
application/config.py<span class="w"> </span><span class="m">14</span><span class="w"> </span><span class="m">14</span><span class="w"> </span><span class="m">0</span>%<span class="w"> </span><span class="m">1</span>-32
application/models.py<span class="w"> </span><span class="m">4</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">100</span>%
-----------------------------------------------------
TOTAL<span class="w"> </span><span class="m">29</span><span class="w"> </span><span class="m">23</span><span class="w"> </span><span class="m">21</span>%
<span class="o">====================================</span><span class="w"> </span>short<span class="w"> </span><span class="nb">test</span><span class="w"> </span>summary<span class="w"> </span><span class="nv">info</span><span class="w"> </span><span class="o">===================================</span>
ERROR<span class="w"> </span>tests/tests/test_user.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!<span class="w"> </span>Interrupted:<span class="w"> </span><span class="m">1</span><span class="w"> </span>error<span class="w"> </span>during<span class="w"> </span>collection<span class="w"> </span>!!!!!!!!!!!!!!!!!!!!!!!!!!!!
<span class="o">=======================================</span><span class="w"> </span><span class="m">1</span><span class="w"> </span>error<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.20s<span class="w"> </span><span class="o">=======================================</span>
Stopping<span class="w"> </span>testing_db_1<span class="w"> </span>...<span class="w"> </span><span class="k">done</span>
Removing<span class="w"> </span>testing_db_1<span class="w"> </span>...<span class="w"> </span><span class="k">done</span>
Removing<span class="w"> </span>network<span class="w"> </span>testing_default
$<span class="w"> </span>
</pre></div> </div> </div><p>I won't show here all the steps of the strict TDD methodology, and implement directly the final solution, which is</p><div class="code"><div class="title"><code>application/models.py</code></div><div class="content"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">flask_sqlalchemy</span> <span class="kn">import</span> <span class="n">SQLAlchemy</span>
<span class="kn">from</span> <span class="nn">flask_migrate</span> <span class="kn">import</span> <span class="n">Migrate</span>
<span class="n">db</span> <span class="o">=</span> <span class="n">SQLAlchemy</span><span class="p">()</span>
<span class="n">migrate</span> <span class="o">=</span> <span class="n">Migrate</span><span class="p">()</span>
<span class="k">class</span> <span class="nc">User</span><span class="p">(</span><span class="n">db</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
<span class="n">__tablename__</span> <span class="o">=</span> <span class="s2">"users"</span>
<span class="nb">id</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">Column</span><span class="p">(</span><span class="n">db</span><span class="o">.</span><span class="n">Integer</span><span class="p">,</span> <span class="n">primary_key</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">email</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">Column</span><span class="p">(</span><span class="n">db</span><span class="o">.</span><span class="n">String</span><span class="p">,</span> <span class="n">unique</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">nullable</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</pre></div> </div> </div><p>With this model the test passes</p><div class="code"><div class="content"><div class="highlight"><pre>$<span class="w"> </span>./manage.py<span class="w"> </span><span class="nb">test</span>
Creating<span class="w"> </span>network<span class="w"> </span><span class="s2">"testing_default"</span><span class="w"> </span>with<span class="w"> </span>the<span class="w"> </span>default<span class="w"> </span>driver
Creating<span class="w"> </span>testing_db_1<span class="w"> </span>...<span class="w"> </span><span class="k">done</span>
<span class="o">===================================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">==================================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.7.5,<span class="w"> </span>pytest-5.4.3,<span class="w"> </span>py-1.9.0,<span class="w"> </span>pluggy-0.13.1<span class="w"> </span>--
/home/leo/devel/flask-tutorial/venv3/bin/python3
cachedir:<span class="w"> </span>.pytest_cache
rootdir:<span class="w"> </span>/home/leo/devel/flask-tutorial
plugins:<span class="w"> </span>flask-1.0.0,<span class="w"> </span>cov-2.10.0
collected<span class="w"> </span><span class="m">1</span><span class="w"> </span>item
tests/test_user.py::test__create_user<span class="w"> </span>PASSED
-----------<span class="w"> </span>coverage:<span class="w"> </span>platform<span class="w"> </span>linux,<span class="w"> </span>python<span class="w"> </span><span class="m">3</span>.7.5-final-0<span class="w"> </span>-----------
Name<span class="w"> </span>Stmts<span class="w"> </span>Miss<span class="w"> </span>Cover<span class="w"> </span>Missing
-----------------------------------------------------
application/app.py<span class="w"> </span><span class="m">11</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">91</span>%<span class="w"> </span><span class="m">19</span>
application/config.py<span class="w"> </span><span class="m">14</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">100</span>%
application/models.py<span class="w"> </span><span class="m">8</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">100</span>%
-----------------------------------------------------
TOTAL<span class="w"> </span><span class="m">33</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">97</span>%
<span class="o">====================================</span><span class="w"> </span><span class="m">1</span><span class="w"> </span>passed<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.14s<span class="w"> </span><span class="o">===================================</span>
Stopping<span class="w"> </span>testing_db_1<span class="w"> </span>...<span class="w"> </span><span class="k">done</span>
Removing<span class="w"> </span>testing_db_1<span class="w"> </span>...<span class="w"> </span><span class="k">done</span>
Removing<span class="w"> </span>network<span class="w"> </span>testing_default
$<span class="w"> </span>
</pre></div> </div> </div><p>Please not that this is a very simple example and that in a real case I would add some other tests before accepting this code. In particular we should check that the field <code>email</code> can be empty, and maybe also test some validation on that field.</p><p>Let's add a very simple route to use the newly created model</p><div class="code"><div class="title"><code>application/app.py</code></div><div class="content"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span>
<span class="kn">from</span> <span class="nn">application.models</span> <span class="kn">import</span> <span class="n">User</span>
<span class="k">def</span> <span class="nf">create_app</span><span class="p">(</span><span class="n">config_name</span><span class="p">):</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
<span class="n">config_module</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"application.config.</span><span class="si">{</span><span class="n">config_name</span><span class="o">.</span><span class="n">capitalize</span><span class="p">()</span><span class="si">}</span><span class="s2">Config"</span>
<span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">from_object</span><span class="p">(</span><span class="n">config_module</span><span class="p">)</span>
<span class="kn">from</span> <span class="nn">application.models</span> <span class="kn">import</span> <span class="n">db</span><span class="p">,</span> <span class="n">migrate</span>
<span class="n">db</span><span class="o">.</span><span class="n">init_app</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
<span class="n">migrate</span><span class="o">.</span><span class="n">init_app</span><span class="p">(</span><span class="n">app</span><span class="p">,</span> <span class="n">db</span><span class="p">)</span>
<span class="nd">@app</span><span class="o">.</span><span class="n">route</span><span class="p">(</span><span class="s2">"/"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">hello_world</span><span class="p">():</span>
<span class="k">return</span> <span class="s2">"Hello, World!"</span>
<span class="hll"> <span class="nd">@app</span><span class="o">.</span><span class="n">route</span><span class="p">(</span><span class="s2">"/users"</span><span class="p">)</span>
</span><span class="hll"> <span class="k">def</span> <span class="nf">users</span><span class="p">():</span>
</span><span class="hll"> <span class="n">num_users</span> <span class="o">=</span> <span class="n">User</span><span class="o">.</span><span class="n">query</span><span class="o">.</span><span class="n">count</span><span class="p">()</span>
</span><span class="hll"> <span class="k">return</span> <span class="sa">f</span><span class="s2">"Number of users: </span><span class="si">{</span><span class="n">num_users</span><span class="si">}</span><span class="s2">"</span>
</span>
<span class="k">return</span> <span class="n">app</span>
</pre></div> </div> </div><p>As you can see I didn't introduce anything too complicated. I import the model <code>User</code> and count the number of entries in its table. We will create the table in a minute with the migration that <code>flask db migrate</code> will create for us, so we expect this to just return a page that says "Number of users: 0", but it's a good demonstration that the connection with the database is working.</p><p>So, let's generate the migration in the database. Spin up the development environment with</p><div class="code"><div class="content"><div class="highlight"><pre>$ ./manage.py compose up -d
</pre></div> </div> </div><p>If this is the first time I spin up the environment I have to create the application database and to initialise the migrations, so I run</p><div class="code"><div class="content"><div class="highlight"><pre>$ ./manage.py create-initial-db
</pre></div> </div> </div><p>As we already initialised Alembic before we don't need to run the command <code>db init</code>. If you do, it will return <code>Error: Directory migrations already exists and is not empty</code>. Now I can create the migration with</p><div class="code"><div class="content"><div class="highlight"><pre>$ ./manage.py flask db migrate -m "Initial user model"
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.autogenerate.compare] Detected added table 'users'
Generating /home/leo/devel/flask-tutorial/migrations/versions/7a09d7f8a8fa_initial_user_model.py ... done
</pre></div> </div> </div><p>As you can see from the output, this created the file <code>migrations/versions/7a09d7f8a8fa_initial_user_model.py</code>. The number <code>7a09d7f8a8fa</code> is just an hex version of a UUID ,so it will be different for you, while the name comes from the commit message. The file itself contains SQLAlchemy code that changes the DB according to the code that we wrote in the application.</p><p>Finally I can apply the migration with</p><div class="code"><div class="content"><div class="highlight"><pre>$ ./manage.py flask db upgrade
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.runtime.migration] Running upgrade -> 7a09d7f8a8fa, Initial user model
</pre></div> </div> </div><p>At this point we can run <code>./manage.py compose exec db psql -U postgres</code> again and see what happened to the database.</p><div class="code"><div class="content"><div class="highlight"><pre>$ ./manage.py compose exec db psql -U postgres
psql (13.0 (Debian 13.0-1.pgdg100+1))
Type "help" for help.
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-------------+----------+----------+------------+------------+-----------------------
application | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
</pre></div> </div> </div><p>You see here that the database <code>application</code> configured with <code>APPLICATION_DB</code> has beed created. You can now connect to it and list the tables</p><div class="code"><div class="content"><div class="highlight"><pre>postgres=# \c application
You are now connected to database "application" as user "postgres".
application=# \dt
List of relations
Schema | Name | Type | Owner
--------+-----------------+-------+----------
public | alembic_version | table | postgres
public | users | table | postgres
(2 rows)
</pre></div> </div> </div><p>The content of the table <code>alembic_version</code> shouldn't be surprising, as it's the UUID used for the migration</p><div class="code"><div class="content"><div class="highlight"><pre>application=# select * from alembic_version;
version_num
--------------
7a09d7f8a8fa
(1 row)
</pre></div> </div> </div><p>The table <code>users</code> contains the fields <code>id</code> and <code>email</code> according to the model that we wrote in Python</p><div class="code"><div class="content"><div class="highlight"><pre>application=# \d users
Table "public.users"
Column | Type | Collation | Nullable | Default
--------+-------------------+-----------+----------+-----------------------------------
id | integer | | not null | nextval('users_id_seq'::regclass)
email | character varying | | not null |
Indexes:
"users_pkey" PRIMARY KEY, btree (id)
"users_email_key" UNIQUE CONSTRAINT, btree (email)
</pre></div> </div> </div><p>You can also open your browser and head to <a href="http://localhost:5000/users">http://localhost:5000/users</a> to see the new route in action. After this we can safely commit my code and move on with the next requirement.</p><h3 id="git-commit-3262">Git commit</h3><p>You can see the changes made in this step through <a href="https://github.com/lgiordani/flask_project_setup/commit/da8c8d98dc8f730b72f99ab88ee6a64b55b23eec">this Git commit</a> or <a href="https://github.com/lgiordani/flask_project_setup/tree/da8c8d98dc8f730b72f99ab88ee6a64b55b23eec">browse the files</a>.</p>
<div class="advertisement">
<a href="https://www.thedigitalcat.academy/freebie-first-class-objects">
<img src="/images/first-class-objects/cover.jpg" />
</a>
<div class="body">
<h2 id="first-class-objects-in-python-fffa">First-class objects in Python<a class="headerlink" href="#first-class-objects-in-python-fffa" title="Permanent link">¶</a></h2>
<p>Higher-order functions, wrappers, and factories</p>
<p>Learn all you need to know to understand first-class citizenship in Python, the gateway to grasp how decorators work and how functional programming can supercharge your code.</p>
<div class="actions">
<a class="action" href="https://www.thedigitalcat.academy/freebie-first-class-objects">Get your FREE copy</a>
</div>
</div>
</div>
<h2 id="final-words-9803">Final words<a class="headerlink" href="#final-words-9803" title="Permanent link">¶</a></h2><p>I hope this post already showed you why a good setup can make the difference. The project is clean and wrapping the command in the management script plus the centralised config proved to be a good choice as it allowed me to solve the problem of migrations and testing in (what I think is) an elegant way. In the next post I'll show you how to easily create scenarios where you can test queries with only specific data in the database. If you find my posts useful please share them with whoever you thing might be interested.</p><h2 id="updates-0083">Updates<a class="headerlink" href="#updates-0083" title="Permanent link">¶</a></h2><p>2020-07-13 <a href="https://github.com/Alladin9393">Vlad Pavlichek</a> found and fixed a typo in the post, where <code>manage.py</code> was missing the extension <code>.py</code>. Thanks Vlad!</p><p>2020-12-22 I reviewed the whole tutorial and corrected several typos</p><h2 id="feedback-d845">Feedback<a class="headerlink" href="#feedback-d845" title="Permanent link">¶</a></h2><p>Feel free to reach me on <a href="https://twitter.com/thedigicat">Twitter</a> if you have questions. The <a href="https://github.com/TheDigitalCatOnline/blog_source/issues">GitHub issues</a> page is the best place to submit corrections.</p>Flask project setup: TDD, Docker, Postgres and more - Part 12020-07-05T13:00:00+01:002021-08-22T10:00:00+00:00Leonardo Giordanitag:www.thedigitalcatonline.com,2020-07-05:/blog/2020/07/05/flask-project-setup-tdd-docker-postgres-and-more-part-1/<p>A step-by-step tutorial on how to setup a Flask project with TDD, Docker and Postgres</p><p>There are tons of tutorials on Internet that tech you how to use a web framework and how to create Web applications, and many of these cover Flask, first of all the impressive <a href="https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world">Flask Mega-Tutorial</a> by Miguel Grinberg (thanks Miguel!). </p><p>Why another tutorial, then? Recently I started working on a small personal project and decided that it was a good chance to refresh my knowledge of the framework. For this reason I temporarily dropped the <a href="https://www.thedigitalcatbooks.com/pycabook-introduction/">clean architecture</a> I often recommend, and started from scratch following some tutorials. My development environment quickly became very messy, and after a while I realised I was very unsatisfied by the global setup.</p><p>So, I decided to start from scratch again, this time writing down some requirements I want from my development setup. I also know very well how complicated the deploy of an application in production can be, so I want my setup to be "deploy-friendly" as much as possible. Having seen too many project suffer from legacy setups, and knowing that many times such issues can be avoided with a minimum amount of planning, I thought this might be interesting for other developers as well. I consider this setup by no means <em>better</em> than others, it simply addresses different concerns.</p><h2 id="what-you-will-learn-5498">What you will learn<a class="headerlink" href="#what-you-will-learn-5498" title="Permanent link">¶</a></h2><p>This post contains a step-by-step description of how I set up a real Flask project that I am working on. It's important that you understand that this is just one of many possible setups, and that my choices are both a matter of personal taste and dictated by some goals that I will state in this section. Changing the requirements would clearly result in a change of the structure. The target of the post is then to show that the setup of a project can take into account many things upfront, without leaving them to an undetermined future when it will likely be too late to tackle them properly.</p><p>The requirements of my setup are the following:</p><ul><li>Use the same database engine in production, in development and for tests</li><li>Run test on an ephemeral database</li><li>Run in production with no changes other that the static configuration</li><li>Have a command to initialise databases and manage migrations</li><li>Have a way to spin up "scenarios" starting from an empty database, to create a sandbox where I can test queries</li><li>Possible simulate production in the local environment</li></ul><p>As for the technologies, I will use Flask, obviously, as the web framework. I will also use Gunicorn as HTTP server (in production) and Postgres for the database part. I won't show here how to create the production infrastructure, but as I work daily with AWS, I will take into account some of its requirements, trying however not to be too committed to a specific solution.</p><h2 id="a-general-advice-9177">A general advice<a class="headerlink" href="#a-general-advice-9177" title="Permanent link">¶</a></h2><p>Proper setup is an investment for the future. As we do in TDD, where we decide to spend time now (writing tests) to avoid spending tenfold later (to find and correct bugs), setting up a project requires time, and might frustrate the desire of "see things happen". Proper setup is a discipline that requires patience and commitment!</p><p>If you are ready to go, join me for this journey towards a great setup of a Flask application.</p><h2 id="the-golden-rule-307e">The golden rule<a class="headerlink" href="#the-golden-rule-307e" title="Permanent link">¶</a></h2><p>The golden rule of any proper infrastructural work is: there has to be a single source of information. The configuration of you project shouldn't be scattered among different files or repositories (not considering secrets, that have to be stored securely). The configuration has to be accessible and easy to convert into different formats to accommodate the needs of different tools. For this reason, the configuration should be stored in a static file format like JSON, YAML, INI, or similar, which can be read and processed by different programming languages and tools.</p><p>My format of choice for this tutorial is JSON, as it can be read by both Python and Terraform, and is natively used by ECS on AWS.</p><h2 id="step-1---requirements-and-editor-2d7a">Step 1 - Requirements and editor<a class="headerlink" href="#step-1---requirements-and-editor-2d7a" title="Permanent link">¶</a></h2><p>My standard structure for Python requirements uses 3 files: <code>production.txt</code>, <code>development.txt</code>, and <code>testing.txt</code>. They are all stored in the same directory called <code>requirements</code>, and are hierarchically connected. </p><div class="code"><div class="title"><code>requirements/production.txt</code></div><div class="content"><div class="highlight"><pre>## This file is currently empty
</pre></div> </div> </div><div class="code"><div class="title"><code>requirements/testing.txt</code></div><div class="content"><div class="highlight"><pre>-r production.txt
</pre></div> </div> </div><div class="code"><div class="title"><code>requirements/development.txt</code></div><div class="content"><div class="highlight"><pre>-r testing.txt
</pre></div> </div> </div><p>There is also a final file <code>requirements.txt</code> that points to the production one.</p><div class="code"><div class="title"><code>requirements.txt</code></div><div class="content"><div class="highlight"><pre>-r requirements/production.txt
</pre></div> </div> </div><p>As you can see this allows me to separate the requirements to avoid installing unneeded packages, which greatly speeds up the deploy in production and keeps things as essential as possible. Production contains the minimum requirements needed to run the project, testing adds to those the packages used to test the code, and development adds to the latter the tools needed during development. A minor shortcoming of this setup is that I might not need in development everything I need in production, for example the HTTP server. I don't think this is significantly affecting my local setup, though, and if I have to decide between production and development, I prefer to keep the former lean and tidy.</p><p>I have my linters already installed system-wide, but as I'm using black to format the code I have to configure flake8 to accept what I'm doing</p><div class="code"><div class="title"><code>.flake8</code></div><div class="content"><div class="highlight"><pre>[flake8]
# Recommend matching the black line length (default 88),
# rather than using the flake8 default of 79:
max-line-length = 100
ignore = E231
</pre></div> </div> </div><p>This is clearly a very personal choice, and you might have different requirements. Take your time to properly configure the editor and the linter(s). Remember that the editor for a programmer is like the violin for the violinist. You need to know it, and to take care of it. So, set it up properly.</p><p>At this point I also create my virtual environment and activate it.</p><h3 id="git-commit-3262">Git commit</h3><p>You can see the changes made in this step through <a href="https://github.com/lgiordani/flask_project_setup/commit/a6c8e7acde7d5d5d89fad22224fff707d625ebe3">this Git commit</a> or <a href="https://github.com/lgiordani/flask_project_setup/tree/a6c8e7acde7d5d5d89fad22224fff707d625ebe3">browse the files</a>.</p><h3 id="resources-9e89">Resources</h3><ul><li><a href="https://packaging.python.org/tutorials/installing-packages/">Installing packages in Python</a></li><li><a href="https://flake8.pycqa.org/en/latest/">flake8</a> - A tool for style guide enforcement</li><li><a href="https://github.com/psf/black">black</a> - The uncompromising Python code formatter</li></ul><h2 id="step-2---flask-project-boilerplate-7d23">Step 2 - Flask project boilerplate<a class="headerlink" href="#step-2---flask-project-boilerplate-7d23" title="Permanent link">¶</a></h2><p>As this will be a Flask application the first thing to do is to install Flask itself. That goes in the production requirements, as that is needed at every stage.</p><div class="code"><div class="title"><code>requirements/production.txt</code></div><div class="content"><div class="highlight"><pre>Flask
</pre></div> </div> </div><p>Now, install the development requirements with</p><div class="code"><div class="content"><div class="highlight"><pre>$<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>-r<span class="w"> </span>requirements/development.txt
</pre></div> </div> </div><p>As we saw before, that file automatically installs the testing and production requirements as well.</p><p>Then we need a directory where to keep all the code that is directly connected with the Flask framework, and where we will start creating the configuration for the application. Create the directory <code>application</code> and the file <code>config.py</code> in it.</p><div class="code"><div class="title"><code>application/config.py</code></div><div class="content"><div class="highlight"><pre><span class="k">class</span> <span class="nc">Config</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Base configuration"""</span>
<span class="k">class</span> <span class="nc">ProductionConfig</span><span class="p">(</span><span class="n">Config</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Production configuration"""</span>
<span class="k">class</span> <span class="nc">DevelopmentConfig</span><span class="p">(</span><span class="n">Config</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Development configuration"""</span>
<span class="k">class</span> <span class="nc">TestingConfig</span><span class="p">(</span><span class="n">Config</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Testing configuration"""</span>
<span class="n">TESTING</span> <span class="o">=</span> <span class="kc">True</span>
</pre></div> </div> </div><p>There are many ways to configure a Flask application, one of which is using Python objects. This allows me to leverage inheritance to avoid duplication (which is always good), so it's my method of choice.</p><p>It's important to understand the variables and the parameters involved in the configuration. As the documentation clearly states, <code>FLASK_ENV</code> and <code>FLASK_DEBUG</code> have to be initialised outside the application as the code might misbehave if they are changed once the engine has been started. Furthermore the variable <code>FLASK_ENV</code> can have only the two values <code>development</code> and <code>production</code>, and the main difference is in performances. The most important thing we need to be aware of is that if <code>FLASK_ENV</code> is <code>development</code>, then <code>FLASK_DEBUG</code> becomes automatically <code>True</code>. To sum up we have the following guidelines:</p><ul><li>It's pointless to set <code>DEBUG</code> and <code>ENV</code> in the application configuration, they have to be environment variables.</li><li>Generally you don't need to set <code>FLASK_DEBUG</code>, just set <code>FLASK_ENV</code> to <code>development</code>.</li><li>Testing doesn't need the debug server turned on, so you can set <code>FLASK_ENV</code> to <code>production</code> during that phase. It needs <code>TESTING</code> set to <code>True</code>, though, and that has to be done inside the application.</li></ul><p>We need now to create the application and to properly configure it.</p><div class="code"><div class="title"><code>application/app.py</code></div><div class="content"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span>
<span class="k">def</span> <span class="nf">create_app</span><span class="p">(</span><span class="n">config_name</span><span class="p">):</span> <span class="callout">1</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
<span class="n">config_module</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"application.config.</span><span class="si">{</span><span class="n">config_name</span><span class="o">.</span><span class="n">capitalize</span><span class="p">()</span><span class="si">}</span><span class="s2">Config"</span> <span class="callout">2</span>
<span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">from_object</span><span class="p">(</span><span class="n">config_module</span><span class="p">)</span>
<span class="nd">@app</span><span class="o">.</span><span class="n">route</span><span class="p">(</span><span class="s2">"/"</span><span class="p">)</span> <span class="callout">3</span>
<span class="k">def</span> <span class="nf">hello_world</span><span class="p">():</span>
<span class="k">return</span> <span class="s2">"Hello, World!"</span>
<span class="k">return</span> <span class="n">app</span>
</pre></div> </div> </div><p>I decided to use an application factory <span class="callout">1</span> that accepts a string <code>config_name</code> that is then converted into the name of the config object <span class="callout">2</span>. For example, if <code>config_name</code> is <code>development</code> the variable <code>config_module</code> becomes <code>application.config.DevelopmentConfig</code> so that <code>app.config.from_object</code> can import it. I also added the standard "Hello, world!" route <span class="callout">3</span> to have a quick way to see if the server is working or not.</p><p>Last, we need something that initializes the application running the application factory and passing the correct value for the parameter <code>config_name</code>. The Flask development server can automatically use any file named <code>wsgi.py</code> in the root directory, and since WSGI is a standard specification using that makes me sure that any HTTP server we will use in production (for example Gunicorn or uWSGI) will be immediately working.</p><div class="code"><div class="title"><code>wsgi.py</code></div><div class="content"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">from</span> <span class="nn">application.app</span> <span class="kn">import</span> <span class="n">create_app</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">create_app</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"FLASK_CONFIG"</span><span class="p">])</span>
</pre></div> </div> </div><p>Here, I decided to read the value of <code>config_name</code> from the variable <code>FLASK_CONFIG</code>. This is not a variable requested by the framework, but I decided to use the prefix <code>FLASK_</code> anyway because it is tightly connected with the structure of the Flask application.</p><p>At this point we can happily run the Flask development server with</p><div class="code"><div class="content"><div class="highlight"><pre>$<span class="w"> </span><span class="nv">FLASK_CONFIG</span><span class="o">=</span><span class="s2">"development"</span><span class="w"> </span>flask<span class="w"> </span>run
<span class="w"> </span>*<span class="w"> </span>Environment:<span class="w"> </span>production
<span class="w"> </span>WARNING:<span class="w"> </span>This<span class="w"> </span>is<span class="w"> </span>a<span class="w"> </span>development<span class="w"> </span>server.<span class="w"> </span>Do<span class="w"> </span>not<span class="w"> </span>use<span class="w"> </span>it<span class="w"> </span><span class="k">in</span><span class="w"> </span>a<span class="w"> </span>production<span class="w"> </span>deployment.
<span class="w"> </span>Use<span class="w"> </span>a<span class="w"> </span>production<span class="w"> </span>WSGI<span class="w"> </span>server<span class="w"> </span>instead.
<span class="w"> </span>*<span class="w"> </span>Debug<span class="w"> </span>mode:<span class="w"> </span>off
<span class="w"> </span>*<span class="w"> </span>Running<span class="w"> </span>on<span class="w"> </span>http://127.0.0.1:5000/<span class="w"> </span><span class="o">(</span>Press<span class="w"> </span>CTRL+C<span class="w"> </span>to<span class="w"> </span>quit<span class="o">)</span>
</pre></div> </div> </div><p>Please note that it says <code>Environment: production</code> because we haven't configured <code>FLASK_ENV</code> yet. If you head to <a href="http://127.0.0.1:5000/">http://127.0.0.1:5000/</a> with your browser you can see the greetings message.</p><h3 id="git-commit-3262">Git commit</h3><p>You can see the changes made in this step through <a href="https://github.com/lgiordani/flask_project_setup/commit/656621980f6f2c2aac3c526b37dca6ac32363bd5">this Git commit</a> or <a href="https://github.com/lgiordani/flask_project_setup/tree/656621980f6f2c2aac3c526b37dca6ac32363bd5">browse the files</a>.</p><h3 id="resources-9e89">Resources</h3><ul><li><a href="https://flask.palletsprojects.com/en/1.1.x/config/">Flask configuration documentation</a></li><li><a href="https://flask.palletsprojects.com/en/1.1.x/patterns/appfactories/">Flask application factories</a></li><li><a href="https://wsgi.readthedocs.io/en/latest/what.html">WSGI</a> - The Python Web Server Gateway Interface</li><li>My post <a href="https://www.thedigitalcatonline.com/blog/2020/02/16/dissecting-a-web-stack/">Dissecting a web stack</a> includes a section on WSGI</li></ul>
<div class="advertisement">
<a href="https://www.thedigitalcat.academy/freebie-first-class-objects">
<img src="/images/first-class-objects/cover.jpg" />
</a>
<div class="body">
<h2 id="first-class-objects-in-python-fffa">First-class objects in Python<a class="headerlink" href="#first-class-objects-in-python-fffa" title="Permanent link">¶</a></h2>
<p>Higher-order functions, wrappers, and factories</p>
<p>Learn all you need to know to understand first-class citizenship in Python, the gateway to grasp how decorators work and how functional programming can supercharge your code.</p>
<div class="actions">
<a class="action" href="https://www.thedigitalcat.academy/freebie-first-class-objects">Get your FREE copy</a>
</div>
</div>
</div>
<h2 id="step-3---application-configuration-b6e8">Step 3 - Application configuration<a class="headerlink" href="#step-3---application-configuration-b6e8" title="Permanent link">¶</a></h2><p>As I mentioned in the introduction, I am going to use a static JSON configuration file. The choice of JSON comes from the fact that it is a widespread file format, accessible from many programming languages, included Terraform, which I plan to use to create my production infrastructure.</p><div class="code"><div class="title"><code>config/development.json</code></div><div class="content"><div class="highlight"><pre><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"FLASK_ENV"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"development"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"FLASK_CONFIG"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"development"</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">]</span>
</pre></div> </div> </div><p>I obviously need a script that extracts variables from the JSON file and converts them into environment variables, so it's time to start writing my own <code>manage.py</code> file. This is a pretty standard concept in the world of Python web frameworks, a tradition initiated by Django. The idea is to centralise all the management functions like starting/stopping the development server or managing database migrations. As in flask this is partially done by the command <code>flask</code> itself, for the time being I just need to wrap it providing suitable environment variables.</p><div class="code"><div class="title"><code>manage.py</code></div><div class="content"><div class="highlight"><pre><span class="ch">#! /usr/bin/env python</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">signal</span>
<span class="kn">import</span> <span class="nn">subprocess</span>
<span class="kn">import</span> <span class="nn">click</span>
<span class="c1"># Ensure an environment variable exists and has a value</span>
<span class="k">def</span> <span class="nf">setenv</span><span class="p">(</span><span class="n">variable</span><span class="p">,</span> <span class="n">default</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="n">variable</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="n">variable</span><span class="p">,</span> <span class="n">default</span><span class="p">)</span>
<span class="n">setenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">,</span> <span class="s2">"development"</span><span class="p">)</span> <span class="callout">1</span>
<span class="c1"># Read configuration from the relative JSON file</span>
<span class="n">config_json_filename</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">)</span> <span class="o">+</span> <span class="s2">".json"</span> <span class="callout">2</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s2">"config"</span><span class="p">,</span> <span class="n">config_json_filename</span><span class="p">))</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">config</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="c1"># Convert the config into a usable Python dictionary</span>
<span class="n">config</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">((</span><span class="n">i</span><span class="p">[</span><span class="s2">"name"</span><span class="p">],</span> <span class="n">i</span><span class="p">[</span><span class="s2">"value"</span><span class="p">])</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">config</span><span class="p">)</span>
<span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">config</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="n">setenv</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">group</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">cli</span><span class="p">():</span>
<span class="k">pass</span>
<span class="nd">@cli</span><span class="o">.</span><span class="n">command</span><span class="p">(</span><span class="n">context_settings</span><span class="o">=</span><span class="p">{</span><span class="s2">"ignore_unknown_options"</span><span class="p">:</span> <span class="kc">True</span><span class="p">})</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">argument</span><span class="p">(</span><span class="s2">"subcommand"</span><span class="p">,</span> <span class="n">nargs</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="n">click</span><span class="o">.</span><span class="n">Path</span><span class="p">())</span>
<span class="k">def</span> <span class="nf">flask</span><span class="p">(</span><span class="n">subcommand</span><span class="p">):</span> <span class="callout">3</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"flask"</span><span class="p">]</span> <span class="o">+</span> <span class="nb">list</span><span class="p">(</span><span class="n">subcommand</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">KeyboardInterrupt</span><span class="p">:</span>
<span class="n">p</span><span class="o">.</span><span class="n">send_signal</span><span class="p">(</span><span class="n">signal</span><span class="o">.</span><span class="n">SIGINT</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="n">cli</span><span class="o">.</span><span class="n">add_command</span><span class="p">(</span><span class="n">flask</span><span class="p">)</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">cli</span><span class="p">()</span>
</pre></div> </div> </div><p>Remember to make the script executable with</p><div class="code"><div class="content"><div class="highlight"><pre>$<span class="w"> </span>chmod<span class="w"> </span><span class="m">775</span><span class="w"> </span>manage.py
</pre></div> </div> </div><p>As you can see I'm using <code>click</code>, which is the recommended way to implement Flask commands. As I might use it to customise subcommands of the flask main script, I decided to stick to one tool and use it for the script <code>manage.py</code> as well.</p><p>The variable <code>APPLICATION_CONFIG</code> <span class="callout">1</span> is the only one that I need to specify, and its default value is <code>development</code>. From that variable I infer the name of the JSON file with the full configuration <span class="callout">2</span> and load environment variables from that. The function <code>flask</code> <span class="callout">3</span> simply wraps the command <code>flask</code> provided by Flask so that I can run <code>./manage.py flask SUBCOMMAND</code> to run it using the configuration <code>development</code> or <code>APPLICATION_CONFIG="foobar" ./manage.py flask SUBCOMMAND</code> to use the <code>foobar</code> one.</p><p>A clarification, to be sure you don't confuse environment variables with each other:</p><ul><li><code>APPLICATION_CONFIG</code> is strictly related to my project and is used <em>only</em> to load a JSON configuration file with the name specified in the variable itself.</li><li><code>FLASK_CONFIG</code> is used to select the Python object that contains the configuration for the Flask application (see <code>application/app.py</code> and <code>application/config.py</code>). The value of the variable is converted into the name of a class.</li><li><code>FLASK_ENV</code> is a variable used by Flask itself, and its values are dictated by it. See the configuration documentation mentioned in the resources of the previous section.</li></ul><p>Now we can run the development server</p><div class="code"><div class="content"><div class="highlight"><pre>$<span class="w"> </span>./manage.py<span class="w"> </span>flask<span class="w"> </span>run
<span class="w"> </span>*<span class="w"> </span>Environment:<span class="w"> </span>development
<span class="w"> </span>*<span class="w"> </span>Debug<span class="w"> </span>mode:<span class="w"> </span>on
<span class="w"> </span>*<span class="w"> </span>Running<span class="w"> </span>on<span class="w"> </span>http://127.0.0.1:5000/<span class="w"> </span><span class="o">(</span>Press<span class="w"> </span>CTRL+C<span class="w"> </span>to<span class="w"> </span>quit<span class="o">)</span>
<span class="w"> </span>*<span class="w"> </span>Restarting<span class="w"> </span>with<span class="w"> </span>stat
<span class="w"> </span>*<span class="w"> </span>Debugger<span class="w"> </span>is<span class="w"> </span>active!
<span class="w"> </span>*<span class="w"> </span>Debugger<span class="w"> </span>PIN:<span class="w"> </span><span class="m">172</span>-719-201
</pre></div> </div> </div><p>Note that it now says <code>Environment: development</code> because of <code>FLASK_ENV</code> has been set to <code>development</code> in the configuration. As we did before, a quick visit to <a href="http://127.0.0.1:5000/">http://127.0.0.1:5000/</a> shows us that everything is up and running.</p><h3 id="git-commit-3262">Git commit</h3><p>You can see the changes made in this step through <a href="https://github.com/lgiordani/flask_project_setup/commit/8330e792aa55d2903fd4846487c64de12530c0d3">this Git commit</a> or <a href="https://github.com/lgiordani/flask_project_setup/tree/8330e792aa55d2903fd4846487c64de12530c0d3">browse the files</a>.</p><h3 id="resources-b015">Resources:</h3><ul><li><a href="https://docs.djangoproject.com/en/3.0/ref/django-admin/">Django's management script</a></li><li><a href="https://click.palletsprojects.com/en/7.x/">Click</a> - A Python package for creating command line interfaces </li></ul><h2 id="step-4---containers-and-orchestration-9ee8">Step 4 - Containers and orchestration<a class="headerlink" href="#step-4---containers-and-orchestration-9ee8" title="Permanent link">¶</a></h2><p>There is no better way to simplify your development than using Docker.</p><p>There is also no better way to complicate your life than using Docker.</p><p>As you might guess, I have mixed feelings about Docker. Don't get me wrong, Linux containers are an amazing concept, and Docker is very useful. It's also a complex technology that sometimes requires a lot of work to get properly configured. In this case the setup will be pretty simple, but there is a major complication with using a database server that I will describe later.</p><p>Running the application in a Docker container allows me to isolate it and to simulate the way I will run it in production. I will use docker-compose, as I expect to have other containers running in my development setup (at least the database), so I can leverage the fact that the docker-compose configuration file can interpolate environment variables. Once again through the environment variable <code>APPLICATION_CONFIG</code> I will select the correct JSON file, load its values in environment variables and then run the docker-compose file.</p><p>First of all we need an image for the Flask application</p><div class="code"><div class="title"><code>docker/Dockerfile</code></div><div class="content"><div class="highlight"><pre><span class="k">FROM</span><span class="w"> </span><span class="s">python:3</span>
<span class="k">ENV</span><span class="w"> </span>PYTHONUNBUFFERED<span class="w"> </span><span class="m">1</span>
<span class="k">RUN</span><span class="w"> </span>mkdir<span class="w"> </span>/opt/code
<span class="k">RUN</span><span class="w"> </span>mkdir<span class="w"> </span>/opt/requirements
<span class="k">WORKDIR</span><span class="w"> </span><span class="s">/opt/code</span>
<span class="k">ADD</span><span class="w"> </span>requirements<span class="w"> </span>/opt/requirements
<span class="k">RUN</span><span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>-r<span class="w"> </span>/opt/requirements/development.txt
</pre></div> </div> </div><p>As you can see the requirements directory is copied into the image, so that Docker can run the command <code>pip install</code> at creation time. The whole code directory will be mounted live into the image at run time.</p><p>This clearly means that every time we change the development requirements we need to rebuild the image. This is not a complicated process, so I will keep it as a manual process for now. To run the image we can create a configuration file for docker-compose.</p><div class="code"><div class="title"><code>docker/development.yml</code></div><div class="content"><div class="highlight"><pre><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="s">'3.4'</span>
<span class="nt">services</span><span class="p">:</span>
<span class="w"> </span><span class="nt">web</span><span class="p">:</span>
<span class="w"> </span><span class="nt">build</span><span class="p">:</span>
<span class="w"> </span><span class="nt">context</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${PWD}</span>
<span class="w"> </span><span class="nt">dockerfile</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">docker/Dockerfile</span>
<span class="w"> </span><span class="nt">environment</span><span class="p">:</span>
<span class="w"> </span><span class="nt">FLASK_ENV</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${FLASK_ENV}</span>
<span class="w"> </span><span class="nt">FLASK_CONFIG</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${FLASK_CONFIG}</span>
<span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">flask run --host 0.0.0.0</span>
<span class="w"> </span><span class="nt">volumes</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${PWD}:/opt/code</span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"5000:5000"</span>
</pre></div> </div> </div><p>As you can see, the docker-compose configuration file can read environment variables natively. To run it we first need to add docker-compose itself to the development requirements.</p><div class="code"><div class="title"><code>requirements/development.txt</code></div><div class="content"><div class="highlight"><pre>-r testing.txt
docker-compose
</pre></div> </div> </div><p>Install it with <code>pip install -r requirements/development.txt</code>, then build the image with</p><div class="code"><div class="content"><div class="highlight"><pre>$<span class="w"> </span><span class="nv">FLASK_ENV</span><span class="o">=</span><span class="s2">"development"</span><span class="w"> </span><span class="nv">FLASK_CONFIG</span><span class="o">=</span><span class="s2">"development"</span><span class="w"> </span>docker-compose<span class="w"> </span>-f<span class="w"> </span>docker/development.yml<span class="w"> </span>build<span class="w"> </span>web
</pre></div> </div> </div><p>This will take some time, as Docker has to download all the required layers and install the requirements.</p><p>We are explicitly passing environment variables here, as we have not wrapped docker-compose in the manage script yet. Once the image has been build, we can run it with the command <code>up</code></p><div class="code"><div class="content"><div class="highlight"><pre>$<span class="w"> </span><span class="nv">FLASK_ENV</span><span class="o">=</span><span class="s2">"development"</span><span class="w"> </span><span class="nv">FLASK_CONFIG</span><span class="o">=</span><span class="s2">"development"</span><span class="w"> </span>docker-compose<span class="w"> </span>-f<span class="w"> </span>docker/development.yml<span class="w"> </span>up
</pre></div> </div> </div><p>This command gives us the following output</p><div class="code"><div class="content"><div class="highlight"><pre>Creating network "docker_default" with the default driver
Creating docker_web_1 ... done
Attaching to docker_web_1
web_1 | * Environment: development
web_1 | * Debug mode: on
web_1 | * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
web_1 | * Restarting with stat
web_1 | * Debugger is active!
web_1 | * Debugger PIN: 234-361-737
</pre></div> </div> </div><p>You can stop the containers pressing <code>Ctrl-C</code>, which gracefully tears down the system. If you run the command <code>up -d</code> docker-compose will run as a daemon, leaving you the control of the current terminal. If docker-compose is running you can <code>docker ps</code> and you should see an output similar to this</p><div class="code"><div class="content"><div class="highlight"><pre>CONTAINER ID IMAGE COMMAND ... PORTS NAMES
c98f35635625 docker_web "flask run --host 0.…" ... 0.0.0.0:5000->5000/tcp docker_web_1
</pre></div> </div> </div><p>If you need to explore the container you can login directly with </p><div class="code"><div class="content"><div class="highlight"><pre>$<span class="w"> </span>docker<span class="w"> </span><span class="nb">exec</span><span class="w"> </span>-it<span class="w"> </span>docker_web_1<span class="w"> </span>bash
</pre></div> </div> </div><p>or with</p><div class="code"><div class="content"><div class="highlight"><pre>$<span class="w"> </span><span class="nv">FLASK_ENV</span><span class="o">=</span><span class="s2">"development"</span><span class="w"> </span><span class="nv">FLASK_CONFIG</span><span class="o">=</span><span class="s2">"development"</span><span class="w"> </span>docker-compose<span class="w"> </span>-f<span class="w"> </span>docker/development.yml<span class="w"> </span><span class="nb">exec</span><span class="w"> </span>web<span class="w"> </span>bash
</pre></div> </div> </div><p>In either case, you will end up in the directory <code>/opt/code</code> (which is the <code>WORKDIR</code> of the image), where the current directory in the host is mounted.</p><p>To tear down the containers, when running as daemon, you can run</p><div class="code"><div class="content"><div class="highlight"><pre>$<span class="w"> </span><span class="nv">FLASK_ENV</span><span class="o">=</span><span class="s2">"development"</span><span class="w"> </span><span class="nv">FLASK_CONFIG</span><span class="o">=</span><span class="s2">"development"</span><span class="w"> </span>docker-compose<span class="w"> </span>-f<span class="w"> </span>docker/development.yml<span class="w"> </span>down
</pre></div> </div> </div><p>Notice that the server now says <code>Running on http://0.0.0.0:5000/</code>, as the Docker container is using that network interface to communicate with the outside world. Since the ports are mapped, however, you can head to either <a href="http://localhost:5000">http://localhost:5000</a> or <a href="http://0.0.0.0:5000">http://0.0.0.0:5000</a> with your browser.</p><p>To simplify the usage of docker-compose, I want to wrap it in the script <code>manage.py</code>, so that it automatically receives environment variables, as their number is going to increase as soon as we add a database.</p><div class="code"><div class="title"><code>manage.py</code></div><div class="content"><div class="highlight"><pre><span class="ch">#! /usr/bin/env python</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">signal</span>
<span class="kn">import</span> <span class="nn">subprocess</span>
<span class="kn">import</span> <span class="nn">click</span>
<span class="n">docker_compose_file</span> <span class="o">=</span> <span class="s2">"docker/development.yml"</span>
<span class="n">docker_compose_cmdline</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"docker-compose"</span><span class="p">,</span> <span class="s2">"-f"</span><span class="p">,</span> <span class="n">docker_compose_file</span><span class="p">]</span>
<span class="c1"># Ensure an environment variable exists and has a value</span>
<span class="k">def</span> <span class="nf">setenv</span><span class="p">(</span><span class="n">variable</span><span class="p">,</span> <span class="n">default</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="n">variable</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="n">variable</span><span class="p">,</span> <span class="n">default</span><span class="p">)</span>
<span class="n">setenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">,</span> <span class="s2">"development"</span><span class="p">)</span>
<span class="c1"># Read configuration from the relative JSON file</span>
<span class="n">config_json_filename</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"APPLICATION_CONFIG"</span><span class="p">)</span> <span class="o">+</span> <span class="s2">".json"</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s2">"config"</span><span class="p">,</span> <span class="n">config_json_filename</span><span class="p">))</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">config</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="c1"># Convert the config into a usable Python dictionary</span>
<span class="n">config</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">((</span><span class="n">i</span><span class="p">[</span><span class="s2">"name"</span><span class="p">],</span> <span class="n">i</span><span class="p">[</span><span class="s2">"value"</span><span class="p">])</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">config</span><span class="p">)</span>
<span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">config</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="n">setenv</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">group</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">cli</span><span class="p">():</span>
<span class="k">pass</span>
<span class="nd">@cli</span><span class="o">.</span><span class="n">command</span><span class="p">(</span><span class="n">context_settings</span><span class="o">=</span><span class="p">{</span><span class="s2">"ignore_unknown_options"</span><span class="p">:</span> <span class="kc">True</span><span class="p">})</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">argument</span><span class="p">(</span><span class="s2">"subcommand"</span><span class="p">,</span> <span class="n">nargs</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="n">click</span><span class="o">.</span><span class="n">Path</span><span class="p">())</span>
<span class="k">def</span> <span class="nf">flask</span><span class="p">(</span><span class="n">subcommand</span><span class="p">):</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"flask"</span><span class="p">]</span> <span class="o">+</span> <span class="nb">list</span><span class="p">(</span><span class="n">subcommand</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">KeyboardInterrupt</span><span class="p">:</span>
<span class="n">p</span><span class="o">.</span><span class="n">send_signal</span><span class="p">(</span><span class="n">signal</span><span class="o">.</span><span class="n">SIGINT</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="nd">@cli</span><span class="o">.</span><span class="n">command</span><span class="p">(</span><span class="n">context_settings</span><span class="o">=</span><span class="p">{</span><span class="s2">"ignore_unknown_options"</span><span class="p">:</span> <span class="kc">True</span><span class="p">})</span>
<span class="nd">@click</span><span class="o">.</span><span class="n">argument</span><span class="p">(</span><span class="s2">"subcommand"</span><span class="p">,</span> <span class="n">nargs</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="n">click</span><span class="o">.</span><span class="n">Path</span><span class="p">())</span>
<span class="k">def</span> <span class="nf">compose</span><span class="p">(</span><span class="n">subcommand</span><span class="p">):</span>
<span class="n">cmdline</span> <span class="o">=</span> <span class="n">docker_compose_cmdline</span> <span class="o">+</span> <span class="nb">list</span><span class="p">(</span><span class="n">subcommand</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span><span class="n">cmdline</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">KeyboardInterrupt</span><span class="p">:</span>
<span class="n">p</span><span class="o">.</span><span class="n">send_signal</span><span class="p">(</span><span class="n">signal</span><span class="o">.</span><span class="n">SIGINT</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">cli</span><span class="p">()</span>
</pre></div> </div> </div><p>You might have noticed that the two functions <code>flask</code> and <code>compose</code> are basically the same code, but I resisted the temptation to refactor them because I know that the command <code>compose</code> will need some changes as soon as I add a database.</p><p>Now I can run <code>./manage.py compose up -d</code> and <code>./manage.py compose down</code> and have the environment variables automatically passed to the system.</p><h3 id="git-commit-3262">Git commit</h3><p>You can see the changes made in this step through <a href="https://github.com/lgiordani/flask_project_setup/commit/9e08735af6177760cd750230122a507b15c9c112">this Git commit</a> or <a href="https://github.com/lgiordani/flask_project_setup/tree/9e08735af6177760cd750230122a507b15c9c112">browse the files</a>.</p>
<div class="advertisement">
<a href="https://www.thedigitalcat.academy/freebie-first-class-objects">
<img src="/images/first-class-objects/cover.jpg" />
</a>
<div class="body">
<h2 id="first-class-objects-in-python-fffa">First-class objects in Python<a class="headerlink" href="#first-class-objects-in-python-fffa" title="Permanent link">¶</a></h2>
<p>Higher-order functions, wrappers, and factories</p>
<p>Learn all you need to know to understand first-class citizenship in Python, the gateway to grasp how decorators work and how functional programming can supercharge your code.</p>
<div class="actions">
<a class="action" href="https://www.thedigitalcat.academy/freebie-first-class-objects">Get your FREE copy</a>
</div>
</div>
</div>
<h3 id="resources-9e89">Resources</h3><ul><li><a href="https://docs.docker.com/compose/">Docker compose</a> - A tool for defining and running multi-container Docker applications</li><li><a href="https://docs.docker.com/network/">Docker networking</a></li><li><a href="https://docs.python.org/3/library/subprocess.html">Python subprocess module</a></li></ul><h2 id="final-words-9803">Final words<a class="headerlink" href="#final-words-9803" title="Permanent link">¶</a></h2><p>That's enough for this first post. We started from scratch and added some boilerplate code for a Flask project, exploring what environment variables are used by the framework, then we added a configuration system, a management script, and finally we run everything in a Docker container. In the next post I will show you how to add a persistent database to the development setup and how to use an ephemeral one for the tests. If you find my posts useful please share them with whoever you thing might be interested. </p><p>Happy development!</p><h2 id="updates-0083">Updates<a class="headerlink" href="#updates-0083" title="Permanent link">¶</a></h2><p>2020-12-22 I reviewed the whole tutorial and corrected several typos</p><h2 id="feedback-d845">Feedback<a class="headerlink" href="#feedback-d845" title="Permanent link">¶</a></h2><p>Feel free to reach me on <a href="https://twitter.com/thedigicat">Twitter</a> if you have questions. The <a href="https://github.com/TheDigitalCatOnline/blog_source/issues">GitHub issues</a> page is the best place to submit corrections.</p>Dissecting a Web stack2020-02-16T15:00:00+00:002020-10-27T08:30:00+00:00Leonardo Giordanitag:www.thedigitalcatonline.com,2020-02-16:/blog/2020/02/16/dissecting-a-web-stack/<p>A layer-by-layer review of the components of a web stack and the reasons behind them</p><blockquote>
<p>It was gross. They wanted me to dissect a frog.</p>
<p>(Beetlejuice, 1988)</p>
</blockquote>
<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h2>
<p>Having recently worked with young web developers who were exposed for the first time to proper production infrastructure, I received many questions about the various components that one can find in the architecture of a "Web service". These questions clearly expressed the confusion (and sometimes the frustration) of developers who understand how to create endpoints in a high-level language such as Node.js or Python, but were never introduced to the complexity of what happens between the user's browser and their framework of choice. Most of the times they don't know why the framework itself is there in the first place.</p>
<p>The challenge is clear if we just list (in random order), some of the words we use when we discuss (Python) Web development: HTTP, cookies, web server, Websockets, FTP, multi-threaded, reverse proxy, Django, nginx, static files, POST, certificates, framework, Flask, SSL, GET, WSGI, session management, TLS, load balancing, Apache.</p>
<p>In this post, I want to review all the words mentioned above (and a couple more) trying to build a production-ready web service from the ground up. I hope this might help young developers to get the whole picture and to make sense of these "obscure" names that senior developers like me tend to drop in everyday conversations (sometimes arguably out of turn).</p>
<p>As the focus of the post is the global architecture and the reasons behind the presence of specific components, the example service I will use will be a basic HTML web page. The reference language will be Python but the overall discussion applies to any language or framework.</p>
<p>My approach will be that of first stating the rationale and then implementing a possible solution. After this, I will point out missing pieces or unresolved issues and move on with the next layer. At the end of the process, the reader should have a clear picture of why each component has been added to the system.</p>
<h2 id="the-perfect-architecture">The perfect architecture<a class="headerlink" href="#the-perfect-architecture" title="Permanent link">¶</a></h2>
<p>A very important underlying concept of system architectures is that there is no <em>perfect solution</em> devised by some wiser genius, that we just need to apply. Unfortunately, often people mistake design patterns for such a "magic solution". The "Design Patterns" original book, however, states that</p>
<blockquote>
<p>Your design should be specific to the problem at hand but also general enough to address future problems and requirements. You also want to avoid redesign, or at least minimize it.</p>
</blockquote>
<p>And later</p>
<blockquote>
<p>Design patterns make it easier to reuse successful designs and architectures. [...] Design patterns help you choose design alternatives that make a system reusable and avoid alternatives that compromise reusability.</p>
</blockquote>
<p>The authors of the book are discussing Object-oriented Programming, but these sentences can be applied to any architecture. As you can see, we have a "problem at hand" and "design alternatives", which means that the most important thing to understand is the requirements, both the present and future ones. Only with clear requirements in mind, one can effectively design a solution, possibly tapping into the great number of patterns that other designers already devised.</p>
<p>A very last remark. A web stack is a complex beast, made of several components and software packages developed by different programmers with different goals in mind. It is perfectly understandable, then, that such components have some degree of superposition. While the division line between theoretical layers is usually very clear, in practice the separation is often blurry. Expect this a lot, and you will never be lost in a web stack anymore.</p>
<h2 id="some-definitions">Some definitions<a class="headerlink" href="#some-definitions" title="Permanent link">¶</a></h2>
<p>Let's briefly review some of the most important concepts involved in a Web stack, the protocols.</p>
<h3 id="tcpip">TCP/IP<a class="headerlink" href="#tcpip" title="Permanent link">¶</a></h3>
<p>TCP/IP is a network protocol, that is, a <em>set of established rules</em> two computers have to follow to get connected over a physical network to exchange messages. TCP/IP is composed of two different protocols covering two different layers of the OSI stack, namely the Transport (TCP) and the Network (IP) ones. TCP/IP can be implemented on top of any physical interface (Data Link and Physical OSI layers), such as Ethernet and Wireless. Actors in a TCP/IP network are identified by a <em>socket</em>, which is a tuple made of an IP address and a port number.</p>
<p>As far as we are concerned when developing a Web service, however, we need to be aware that TCP/IP is a <em>reliable</em> protocol, which in telecommunications means that the protocol itself takes care or retransmissions when packets get lost. In other words, while the speed of the communication is not granted, we can be sure that once a message is sent it will reach its destination without errors.</p>
<h3 id="http">HTTP<a class="headerlink" href="#http" title="Permanent link">¶</a></h3>
<p>TCP/IP can guarantee that the raw bytes one computer sends will reach their destination, but this leaves completely untouched the problem of how to send meaningful information. In particular, in 1989 the problem Tim Barners-Lee wanted to solve was how to uniquely name hypertext resources in a network and how to access them.</p>
<p>HTTP is the protocol that was devised to solve such a problem and has since greatly evolved. With the help of other protocols such as WebSocket, HTTP invaded areas of communication for which it was originally considered unsuitable such as real-time communication or gaming.</p>
<p>At its core, HTTP is a protocol that states the format of a text request and the possible text responses. The initial version 0.9 published in 1991 defined the concept of URL and allowed only the GET operation that requested a specific resource. HTTP 1.0 and 1.1 added crucial features such as headers, more methods, and important performance optimisations. At the time of writing the adoption of HTTP/2 is around 45% of the websites in the world, and HTTP/3 is still a draft.</p>
<p>The most important feature of HTTP we need to keep in mind as developers is that it is a <em>stateless</em> protocol. This means that the protocol doesn't require the server to keep track of the state of the communication between requests, basically leaving session management to the developer of the service itself.</p>
<p>Session management is crucial nowadays because you usually want to have an authentication layer in front of a service, where a user provides credentials and accesses some private data. It is, however, useful in other contexts such as visual preferences or choices made by the user and re-used in later accesses to the same website. Typical solutions to the session management problem of HTTP involve the use of cookies or session tokens.</p>
<h3 id="https">HTTPS<a class="headerlink" href="#https" title="Permanent link">¶</a></h3>
<p>Security has become a very important word in recent years, and with a reason. The amount of sensitive data we exchange on the Internet or store on digital devices is increasing exponentially, but unfortunately so is the number of malicious attackers and the level of damage they can cause with their actions. The HTTP protocol is inherently</p>
<p>HTTP is inherently insecure, being a plain text communication between two servers that usually happens on a completely untrustable network such as the Internet. While security wasn't an issue when the protocol was initially conceived, it is nowadays a problem of paramount importance, as we exchange private information, often vital for people's security or for businesses. We need to be sure we are sending information to the correct server and that the data we send cannot be intercepted.</p>
<p>HTTPS solves both the problem of tampering and eavesdropping, encrypting HTTP with the Transport Layer Security (TLS) protocol, that also enforces the usage of digital certificates, issued by a trusted authority. At the time of writing, approximately 80% of websites loaded by Firefox use HTTPS by default. When a server receives an HTTPS connection and transforms it into an HTTP one it is usually said that it <em>terminates TLS</em> (or SSL, the old name of TLS).</p>
<h3 id="websocket">WebSocket<a class="headerlink" href="#websocket" title="Permanent link">¶</a></h3>
<p>One great disadvantage of HTTP is that communication is always initiated by the client and that the server can send data only when this is explicitly requested. Polling can be implemented to provide an initial solution, but it cannot guarantee the performances of proper full-duplex communication, where a channel is kept open between server and client and both can send data without being requested. Such a channel is provided by the WebSocket protocol.</p>
<p>WebSocket is a killer technology for applications like online gaming, real-time feeds like financial tickers or sports news, or multimedia communication like conferencing or remote education.</p>
<p>It is important to understand that WebSocket is not HTTP, and can exist without it. It is also true that this new protocol was designed to be used on top of an existing HTTP connection, so a WebSocket communication is often found in parts of a Web page, which was originally retrieved using HTTP in the first place.</p>
<h2 id="implementing-a-service-over-http">Implementing a service over HTTP<a class="headerlink" href="#implementing-a-service-over-http" title="Permanent link">¶</a></h2>
<p>Let's finally start discussing bits and bytes. The starting point for our journey is a service over HTTP, which means there is an HTTP request-response exchange. As an example, let us consider a GET request, the simplest of the HTTP methods.</p>
<div class="highlight"><pre><span></span><code><span class="nf">GET</span> <span class="nn">/</span> <span class="kr">HTTP</span><span class="o">/</span><span class="m">1.1</span>
<span class="na">Host</span><span class="o">:</span> <span class="l">localhost</span>
<span class="na">User-Agent</span><span class="o">:</span> <span class="l">curl/7.65.3</span>
<span class="na">Accept</span><span class="o">:</span> <span class="l">*/*</span>
</code></pre></div>
<p>As you can see, the client is sending a pure text message to the server, with the format specified by the HTTP protocol. The first line contains the method name (<code>GET</code>), the URL (<code>/</code>) and the protocol we are using, including its version (<code>HTTP/1.1</code>). The remaining lines are called <em>headers</em> and contain metadata that can help the server to manage the request. The complete value of the <code>Host</code> header is in this case <code>localhost:80</code>, but as the standard port for HTTP services is 80, we don't need to specify it.</p>
<p>If the server <code>localhost</code> is <em>serving</em> HTTP (i.e. running some software that understands HTTP) on port 80 the response we might get is something similar to</p>
<div class="highlight"><pre><span></span><code><span class="kr">HTTP</span><span class="o">/</span><span class="m">1.0</span> <span class="m">200</span> <span class="ne">OK</span>
<span class="na">Date</span><span class="o">:</span> <span class="l">Mon, 10 Feb 2020 08:41:33 GMT</span>
<span class="na">Content-type</span><span class="o">:</span> <span class="l">text/html</span>
<span class="na">Content-Length</span><span class="o">:</span> <span class="l">26889</span>
<span class="na">Last-Modified</span><span class="o">:</span> <span class="l">Mon, 10 Feb 2020 08:41:27 GMT</span>
<span class="cp"><!DOCTYPE HTML></span>
<span class="p"><</span><span class="nt">html</span><span class="p">></span>
...
<span class="p"></</span><span class="nt">html</span><span class="p">></span>
</code></pre></div>
<p>As happened for the request, the response is a text message, formatted according to the standard. The first line mentions the protocol and the status of the request (<code>200</code> in this case, that means success), while the following lines contain metadata in various headers. Finally, after an empty line, the message contains the resource the client asked for, the source code of the base URL of the website in this case. Since this HTML page probably contains references to other resources like CSS, JS, images, and so on, the browser will send several other requests to gather all the data it needs to properly show the page to the user.</p>
<p>So, the first problem we have is that of implementing a server that understands this protocol and sends a proper response when it receives an HTTP request. We should try to load the requested resource and return either a success (HTTP 200) if we can find it, or a failure (HTTP 404) if we can't.</p>
<h2 id="1-sockets-and-parsers">1 Sockets and parsers<a class="headerlink" href="#1-sockets-and-parsers" title="Permanent link">¶</a></h2>
<h3 id="11-rationale">1.1 Rationale<a class="headerlink" href="#11-rationale" title="Permanent link">¶</a></h3>
<p>TCP/IP is a network protocol that works with <em>sockets</em>. A socket is a tuple of an IP address (unique in the network) and a port (unique for a specific IP address) that the computer uses to communicate with others. A socket is a file-like object in an operating system, that can be thus <em>opened</em> and <em>closed</em>, and that we can <em>read</em> from or <em>write</em> to. Socket programming is a pretty low-level approach to the network, but you need to be aware that every software in your computer that provides network access has ultimately to deal with sockets (most probably through some library, though).</p>
<p>Since we are building things from the ground up, let's implement a small Python program that opens a socket connection, receives an HTTP request, and sends an HTTP response. As port 80 is a "low port" (a number smaller than 1024), we usually don't have permissions to open sockets there, so I will use port 8080. This is not a problem for now, as HTTP can be served on any port.</p>
<h3 id="12-implementation">1.2 Implementation<a class="headerlink" href="#12-implementation" title="Permanent link">¶</a></h3>
<p>Create the file <code>server.py</code> and type this code. Yes, <strong>type it</strong>, don't just copy and paste, you will not learn anything otherwise.</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">socket</span>
<span class="c1">## Create a socket instance</span>
<span class="c1">## AF_INET: use IP protocol version 4</span>
<span class="c1">## SOCK_STREAM: full-duplex byte stream</span>
<span class="n">s</span> <span class="o">=</span> <span class="n">socket</span><span class="o">.</span><span class="n">socket</span><span class="p">(</span><span class="n">socket</span><span class="o">.</span><span class="n">AF_INET</span><span class="p">,</span> <span class="n">socket</span><span class="o">.</span><span class="n">SOCK_STREAM</span><span class="p">)</span>
<span class="c1">## Allow reuse of addresses</span>
<span class="n">s</span><span class="o">.</span><span class="n">setsockopt</span><span class="p">(</span><span class="n">socket</span><span class="o">.</span><span class="n">SOL_SOCKET</span><span class="p">,</span> <span class="n">socket</span><span class="o">.</span><span class="n">SO_REUSEADDR</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="c1">## Bind the socket to any address, port 8080, and listen</span>
<span class="n">s</span><span class="o">.</span><span class="n">bind</span><span class="p">((</span><span class="s1">''</span><span class="p">,</span> <span class="mi">8080</span><span class="p">))</span>
<span class="n">s</span><span class="o">.</span><span class="n">listen</span><span class="p">()</span>
<span class="c1">## Serve forever</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="c1"># Accept the connection</span>
<span class="n">conn</span><span class="p">,</span> <span class="n">addr</span> <span class="o">=</span> <span class="n">s</span><span class="o">.</span><span class="n">accept</span><span class="p">()</span>
<span class="c1"># Receive data from this socket using a buffer of 1024 bytes</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">recv</span><span class="p">(</span><span class="mi">1024</span><span class="p">)</span>
<span class="c1"># Print out the data</span>
<span class="nb">print</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">))</span>
<span class="c1"># Close the connection</span>
<span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</code></pre></div>
<p>This little program accepts a connection on port 8080 and prints the received data on the terminal. You can test it executing it and then running <code>curl localhost:8080</code> in another terminal. You should see something like</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>python3<span class="w"> </span>server.py<span class="w"> </span>
GET<span class="w"> </span>/<span class="w"> </span>HTTP/1.1
Host:<span class="w"> </span>localhost:8080
User-Agent:<span class="w"> </span>curl/7.65.3
Accept:<span class="w"> </span>*/*
</code></pre></div>
<p>The server keeps running the code in the <code>while</code> loop, so if you want to terminate it you have to do it with Ctrl+C. So far so good, but this is not an HTTP server yet, as it sends no response; you should actually receive an error message from curl that says <code>curl: (52) Empty reply from server</code>.</p>
<p>Sending back a standard response is very simple, we just need to call <code>conn.sendall</code> passing the raw bytes. A minimal HTTP response contains the protocol and the status, an empty line, and the actual content, for example</p>
<div class="highlight"><pre><span></span><code><span class="kr">HTTP</span><span class="o">/</span><span class="m">1.1</span> <span class="m">200</span> <span class="ne">OK</span>
Hi there!
</code></pre></div>
<p>Our server becomes then</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">socket</span>
<span class="c1">## Create a socket instance</span>
<span class="c1">## AF_INET: use IP protocol version 4</span>
<span class="c1">## SOCK_STREAM: full-duplex byte stream</span>
<span class="n">s</span> <span class="o">=</span> <span class="n">socket</span><span class="o">.</span><span class="n">socket</span><span class="p">(</span><span class="n">socket</span><span class="o">.</span><span class="n">AF_INET</span><span class="p">,</span> <span class="n">socket</span><span class="o">.</span><span class="n">SOCK_STREAM</span><span class="p">)</span>
<span class="c1">## Allow reuse of addresses</span>
<span class="n">s</span><span class="o">.</span><span class="n">setsockopt</span><span class="p">(</span><span class="n">socket</span><span class="o">.</span><span class="n">SOL_SOCKET</span><span class="p">,</span> <span class="n">socket</span><span class="o">.</span><span class="n">SO_REUSEADDR</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="c1">## Bind the socket to any address, port 8080, and listen</span>
<span class="n">s</span><span class="o">.</span><span class="n">bind</span><span class="p">((</span><span class="s1">''</span><span class="p">,</span> <span class="mi">8080</span><span class="p">))</span>
<span class="n">s</span><span class="o">.</span><span class="n">listen</span><span class="p">()</span>
<span class="c1">## Serve forever</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="c1"># Accept the connection</span>
<span class="n">conn</span><span class="p">,</span> <span class="n">addr</span> <span class="o">=</span> <span class="n">s</span><span class="o">.</span><span class="n">accept</span><span class="p">()</span>
<span class="c1"># Receive data from this socket using a buffer of 1024 bytes</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">recv</span><span class="p">(</span><span class="mi">1024</span><span class="p">)</span>
<span class="c1"># Print out the data</span>
<span class="nb">print</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">))</span>
<span class="n">conn</span><span class="o">.</span><span class="n">sendall</span><span class="p">(</span><span class="nb">bytes</span><span class="p">(</span><span class="s2">"HTTP/1.1 200 OK</span><span class="se">\n\n</span><span class="s2">Hi there!</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s1">'utf-8'</span><span class="p">))</span>
<span class="c1"># Close the connection</span>
<span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</code></pre></div>
<p>At this point, we are not really responding to the user's request, however. Try different curl command lines like <code>curl localhost:8080/index.html</code> or <code>curl localhost:8080/main.css</code> and you will always receive the same response. We should try to find the resource the user is asking for and send that back in the response content.</p>
<p>This version of the HTTP server properly extracts the resource and tries to load it from the current directory, returning either a success of a failure</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">socket</span>
<span class="kn">import</span> <span class="nn">re</span>
<span class="c1">## Create a socket instance</span>
<span class="c1">## AF_INET: use IP protocol version 4</span>
<span class="c1">## SOCK_STREAM: full-duplex byte stream</span>
<span class="n">s</span> <span class="o">=</span> <span class="n">socket</span><span class="o">.</span><span class="n">socket</span><span class="p">(</span><span class="n">socket</span><span class="o">.</span><span class="n">AF_INET</span><span class="p">,</span> <span class="n">socket</span><span class="o">.</span><span class="n">SOCK_STREAM</span><span class="p">)</span>
<span class="c1">## Allow reuse of addresses</span>
<span class="n">s</span><span class="o">.</span><span class="n">setsockopt</span><span class="p">(</span><span class="n">socket</span><span class="o">.</span><span class="n">SOL_SOCKET</span><span class="p">,</span> <span class="n">socket</span><span class="o">.</span><span class="n">SO_REUSEADDR</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="c1">## Bind the socket to any address, port 8080, and listen</span>
<span class="n">s</span><span class="o">.</span><span class="n">bind</span><span class="p">((</span><span class="s1">''</span><span class="p">,</span> <span class="mi">8080</span><span class="p">))</span>
<span class="n">s</span><span class="o">.</span><span class="n">listen</span><span class="p">()</span>
<span class="n">HEAD_200</span> <span class="o">=</span> <span class="s2">"HTTP/1.1 200 OK</span><span class="se">\n\n</span><span class="s2">"</span>
<span class="n">HEAD_404</span> <span class="o">=</span> <span class="s2">"HTTP/1.1 404 Not Found</span><span class="se">\n\n</span><span class="s2">"</span>
<span class="c1">## Serve forever</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="c1"># Accept the connection</span>
<span class="n">conn</span><span class="p">,</span> <span class="n">addr</span> <span class="o">=</span> <span class="n">s</span><span class="o">.</span><span class="n">accept</span><span class="p">()</span>
<span class="c1"># Receive data from this socket using a buffer of 1024 bytes</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">recv</span><span class="p">(</span><span class="mi">1024</span><span class="p">)</span>
<span class="n">request</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">)</span>
<span class="c1"># Print out the data</span>
<span class="nb">print</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
<span class="n">resource</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="sa">r</span><span class="s1">'GET /(.*) HTTP'</span><span class="p">,</span> <span class="n">request</span><span class="p">)</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">resource</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">content</span> <span class="o">=</span> <span class="n">HEAD_200</span> <span class="o">+</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Resource </span><span class="si">{}</span><span class="s1"> correctly served'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">resource</span><span class="p">))</span>
<span class="k">except</span> <span class="ne">FileNotFoundError</span><span class="p">:</span>
<span class="n">content</span> <span class="o">=</span> <span class="n">HEAD_404</span> <span class="o">+</span> <span class="s2">"Resource /</span><span class="si">{}</span><span class="s2"> cannot be found</span><span class="se">\n</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">resource</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Resource </span><span class="si">{}</span><span class="s1"> cannot be loaded'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">resource</span><span class="p">))</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'--------------------'</span><span class="p">)</span>
<span class="n">conn</span><span class="o">.</span><span class="n">sendall</span><span class="p">(</span><span class="nb">bytes</span><span class="p">(</span><span class="n">content</span><span class="p">,</span> <span class="s1">'utf-8'</span><span class="p">))</span>
<span class="c1"># Close the connection</span>
<span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</code></pre></div>
<p>As you can see this implementation is extremely simple. If you create a simple local file named <code>index.html</code> with this content</p>
<div class="highlight"><pre><span></span><code><span class="p"><</span><span class="nt">head</span><span class="p">></span>
<span class="p"><</span><span class="nt">title</span><span class="p">></span>This is my page<span class="p"></</span><span class="nt">title</span><span class="p">></span>
<span class="p"><</span><span class="nt">link</span> <span class="na">rel</span><span class="o">=</span><span class="s">"stylesheet"</span> <span class="na">href</span><span class="o">=</span><span class="s">"main.css"</span><span class="p">></span>
<span class="p"></</span><span class="nt">head</span><span class="p">></span>
<span class="p"><</span><span class="nt">html</span><span class="p">></span>
<span class="p"><</span><span class="nt">p</span><span class="p">></span>Some random content<span class="p"></</span><span class="nt">p</span><span class="p">></span>
<span class="p"></</span><span class="nt">html</span><span class="p">></span>
</code></pre></div>
<p>and run <code>curl localhost:8080/index.html</code> you will see the content of the file. At this point, you can even use your browser to open <code>http://localhost:8080/index.html</code> and you will see the title of the page and the content. A Web browser is a software capable of sending HTTP requests and of interpreting the content of the responses if this is HTML (and many other file types like images or videos), so it can <em>render</em> the content of the message. The browser is also responsible of retrieving the missing resources needed for the rendering, so when you provide links to style sheets or JS scripts with the <code><link></code> or the <code><script></code> tags in the HTML code of a page, you are instructing the browser to send an HTTP GET request for those files as well.</p>
<p>The output of <code>server.py</code> when I access <code>http://localhost:8080/index.html</code> is</p>
<div class="highlight"><pre><span></span><code><span class="nf">GET</span> <span class="nn">/index.html</span> <span class="kr">HTTP</span><span class="o">/</span><span class="m">1.1</span>
<span class="na">Host</span><span class="o">:</span> <span class="l">localhost:8080</span>
<span class="na">User-Agent</span><span class="o">:</span> <span class="l">Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0</span>
<span class="na">Accept</span><span class="o">:</span> <span class="l">text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8</span>
<span class="na">Accept-Language</span><span class="o">:</span> <span class="l">en-GB,en;q=0.5</span>
<span class="na">Accept-Encoding</span><span class="o">:</span> <span class="l">gzip, deflate</span>
<span class="na">Connection</span><span class="o">:</span> <span class="l">keep-alive</span>
<span class="na">Upgrade-Insecure-Requests</span><span class="o">:</span> <span class="l">1</span>
<span class="na">Pragma</span><span class="o">:</span> <span class="l">no-cache</span>
<span class="na">Cache-Control</span><span class="o">:</span> <span class="l">no-cache</span>
Resource index.html correctly served
--------------------
GET /main.css HTTP/1.1
Host: localhost:8080
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0
Accept: text/css,*/*;q=0.1
Accept-Language: en-GB,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Referer: http://localhost:8080/index.html
Pragma: no-cache
Cache-Control: no-cache
Resource main.css cannot be loaded
--------------------
GET /favicon.ico HTTP/1.1
Host: localhost:8080
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0
Accept: image/webp,*/*
Accept-Language: en-GB,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Resource favicon.ico cannot be loaded
--------------------
</code></pre></div>
<p>As you can see the browser sends rich HTTP requests, with a lot of headers, automatically requesting the CSS file mentioned in the HTML code and automatically trying to retrieve a favicon image.</p>
<h3 id="13-resources">1.3 Resources<a class="headerlink" href="#13-resources" title="Permanent link">¶</a></h3>
<p>These resources provide more detailed information on the topics discussed in this section</p>
<ul>
<li><a href="https://docs.python.org/3/howto/sockets.html">Python 3 Socket Programming HOWTO</a></li>
<li><a href="https://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5">HTTP/1.1 Request format</a></li>
<li><a href="https://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html#sec6">HTTP/1.1 Response format</a></li>
<li>The source code of this example is available <a href="https://github.com/lgiordani/dissecting-a-web-stack-code/tree/master/1_sockets_and_parsers">here</a></li>
</ul>
<h3 id="14-issues">1.4 Issues<a class="headerlink" href="#14-issues" title="Permanent link">¶</a></h3>
<p>It gives a certain dose of satisfaction to build something from scratch and discover that it works smoothly with full-fledged software like the browser you use every day. I also think it is very interesting to discover that technologies like HTTP, that basically run the world nowadays, are at their core very simple.</p>
<p>That said, there are many features of HTTP that we didn't cover with our simple socket programming. For starters, HTTP/1.0 introduced other methods after GET, such as POST that is of paramount importance for today's websites, where users keep sending information to servers through forms. To implement all 9 HTTP methods we need to properly parse the incoming request and add relevant functions to our code.</p>
<p>At this point, however, you might notice that we are dealing a lot with low-level details of the protocol, which is usually not the core of our business. When we build a service over HTTP we believe that we have the knowledge to properly implement some code that can simplify a certain process, be it searching for other websites, shopping for books or sharing pictures with friends. We don't want to spend our time understanding the subtleties of the TCP/IP sockets and writing parsers for request-response protocols. It is nice to see how these technologies work, but on a daily basis, we need to focus on something at a higher level.</p>
<p>The situation of our small HTTP server is possibly worsened by the fact that HTTP is a stateless protocol. The protocol doesn't provide any way to connect two successive requests, thus keeping track of the <em>state</em> of the communication, which is the cornerstone of modern Internet. Every time we authenticate on a website and we want to visit other pages we need the server to remember who we are, and this implies keeping track of the state of the connection.</p>
<p>Long story short: to work as a proper HTTP server, our code should at this point implement all HTTP methods and cookies management. We also need to support other protocols like Websockets. These are all but trivial tasks, so we definitely need to add some component to the whole system that lets us focus on the business logic and not on the low-level details of application protocols.</p>
<h2 id="2-web-framework">2 Web framework<a class="headerlink" href="#2-web-framework" title="Permanent link">¶</a></h2>
<h3 id="21-rationale">2.1 Rationale<a class="headerlink" href="#21-rationale" title="Permanent link">¶</a></h3>
<p>Enter the Web framework!</p>
<p>As I discussed many times (see <a href="https://www.thedigitalcatonline.com/blog/2018/12/20/cabook/">the book on clean architectures</a> or <a href="https://www.thedigitalcatonline.com/blog/2016/11/14/clean-architectures-in-python-a-step-by-step-example/">the relative post</a>) the role of the Web framework is that of <em>converting HTTP requests into function calls</em>, and function return values into HTTP responses. The framework's true nature is that of a layer that connects a working business logic to the Web, through HTTP and related protocols. The framework takes care of session management for us and maps URLs to functions, allowing us to focus on the application logic.</p>
<p>In the grand scheme of an HTTP service, this is what the framework is supposed to do. Everything the framework provides out of this scope, like layers to access DBs, template engines, and interfaces to other systems, is an addition that you, as a programmer, may find useful, but is not in principle part of the reason why we added the framework to the system. We add the framework because it acts as a layer between our business logic and HTTP.</p>
<h3 id="22-implementation">2.2 Implementation<a class="headerlink" href="#22-implementation" title="Permanent link">¶</a></h3>
<p>Thanks to Miguel Gringberg and his <a href="https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world">amazing Flask mega-tutorial</a> I can set up Flask in seconds. I will not run through the tutorial here, as you can follow it on Miguel's website. I will only use the content of the first article (out of 23!) to create an extremely simple "Hello, world" application.</p>
<p>To run the following example you will need a virtual environment and you will have to <code>pip install flask</code>. Follow Miguel's tutorial if you need more details on this.</p>
<p>The <code>app/__init__.py</code> file is</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span>
<span class="n">application</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
<span class="kn">from</span> <span class="nn">app</span> <span class="kn">import</span> <span class="n">routes</span>
</code></pre></div>
<p>and the <code>app/routes.py</code> file is</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">app</span> <span class="kn">import</span> <span class="n">application</span>
<span class="nd">@application</span><span class="o">.</span><span class="n">route</span><span class="p">(</span><span class="s1">'/'</span><span class="p">)</span>
<span class="nd">@application</span><span class="o">.</span><span class="n">route</span><span class="p">(</span><span class="s1">'/index'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">index</span><span class="p">():</span>
<span class="k">return</span> <span class="s2">"Hello, world!"</span>
</code></pre></div>
<p>You can already see here the power of a framework in action. We defined an <code>index</code> function and connected it with two different URLs (<code>/</code> and <code>/index</code>) in 3 lines of Python. This leaves us time and energy to properly work on the business logic, that in this case is a revolutionary "Hello, world!". Nobody ever did this before.</p>
<p>Finally, the <code>service.py</code> file is</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">app</span> <span class="kn">import</span> <span class="n">application</span>
</code></pre></div>
<p>Flask comes with a so-called development web server (do these words ring any bell now?) that we can run on a terminal</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nv">FLASK_APP</span><span class="o">=</span>service.py<span class="w"> </span>flask<span class="w"> </span>run
<span class="w"> </span>*<span class="w"> </span>Serving<span class="w"> </span>Flask<span class="w"> </span>app<span class="w"> </span><span class="s2">"service.py"</span>
<span class="w"> </span>*<span class="w"> </span>Environment:<span class="w"> </span>production
<span class="w"> </span>WARNING:<span class="w"> </span>This<span class="w"> </span>is<span class="w"> </span>a<span class="w"> </span>development<span class="w"> </span>server.<span class="w"> </span>Do<span class="w"> </span>not<span class="w"> </span>use<span class="w"> </span>it<span class="w"> </span><span class="k">in</span><span class="w"> </span>a<span class="w"> </span>production<span class="w"> </span>deployment.
<span class="w"> </span>Use<span class="w"> </span>a<span class="w"> </span>production<span class="w"> </span>WSGI<span class="w"> </span>server<span class="w"> </span>instead.
<span class="w"> </span>*<span class="w"> </span>Debug<span class="w"> </span>mode:<span class="w"> </span>off
<span class="w"> </span>*<span class="w"> </span>Running<span class="w"> </span>on<span class="w"> </span>http://127.0.0.1:5000/<span class="w"> </span><span class="o">(</span>Press<span class="w"> </span>CTRL+C<span class="w"> </span>to<span class="w"> </span>quit<span class="o">)</span>
</code></pre></div>
<p>You can now visit the given URL with your browser and see that everything works properly. Remember that 127.0.0.1 is the special IP address that refers to "this computer"; the name <code>localhost</code> is usually created by the operating system as an alias for that, so the two are interchangeable. As you can see the standard port for Flask's development server is 5000, so you have to mention it explicitly, otherwise your browser would try to access port 80 (the default HTTP one). When you connect with the browser you will see some log messages about the HTTP requests</p>
<div class="highlight"><pre><span></span><code><span class="m">127</span>.0.0.1<span class="w"> </span>-<span class="w"> </span>-<span class="w"> </span><span class="o">[</span><span class="m">14</span>/Feb/2020<span class="w"> </span><span class="m">14</span>:54:27<span class="o">]</span><span class="w"> </span><span class="s2">"GET / HTTP/1.1"</span><span class="w"> </span><span class="m">200</span><span class="w"> </span>-
<span class="m">127</span>.0.0.1<span class="w"> </span>-<span class="w"> </span>-<span class="w"> </span><span class="o">[</span><span class="m">14</span>/Feb/2020<span class="w"> </span><span class="m">14</span>:54:28<span class="o">]</span><span class="w"> </span><span class="s2">"GET /favicon.ico HTTP/1.1"</span><span class="w"> </span><span class="m">404</span><span class="w"> </span>-
</code></pre></div>
<p>You can recognise both now, as those are the same request we got with our little server in the previous part of the article.</p>
<h3 id="23-resources">2.3 Resources<a class="headerlink" href="#23-resources" title="Permanent link">¶</a></h3>
<p>These resources provide more detailed information on the topics discussed in this section</p>
<ul>
<li><a href="https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world">Miguel Gringberg's amazing Flask mega-tutorial</a></li>
<li><a href="https://en.wikipedia.org/wiki/Localhost">What is localhost</a></li>
<li>The source code of this example is available <a href="https://github.com/lgiordani/dissecting-a-web-stack-code/tree/master/2_web_framework">here</a></li>
</ul>
<h3 id="24-issues">2.4 Issues<a class="headerlink" href="#24-issues" title="Permanent link">¶</a></h3>
<p>Apparently, we solved all our problems, and many programmers just stop here. They learn how to use the framework (which is a big achievement!), but as we will shortly discover, this is not enough for a production system. Let's have a closer look at the output of the Flask server. It clearly says, among other things</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>WARNING:<span class="w"> </span>This<span class="w"> </span>is<span class="w"> </span>a<span class="w"> </span>development<span class="w"> </span>server.<span class="w"> </span>Do<span class="w"> </span>not<span class="w"> </span>use<span class="w"> </span>it<span class="w"> </span><span class="k">in</span><span class="w"> </span>a<span class="w"> </span>production<span class="w"> </span>deployment.
<span class="w"> </span>Use<span class="w"> </span>a<span class="w"> </span>production<span class="w"> </span>WSGI<span class="w"> </span>server<span class="w"> </span>instead.
</code></pre></div>
<p>The main issue we have when we deal with any production system is represented by performances. Think about what we do with JavaScript when we minimise the code: we consciously obfuscate the code in order to make the file smaller, but this is done for the sole purpose of making the file faster to retrieve.</p>
<p>For HTTP servers the story is not very different. The Web framework usually provides a development Web server, as Flask does, which properly implements HTTP, but does it in a very inefficient way. For starters, this is a <em>blocking</em> framework, which means that if our request takes seconds to be served (for example because the endpoint retrieves data from a very slow database), any other request will have to wait to be served in a queue. That ultimately means that the user will see a spinner in the browser's tab and just shake their head thinking that we can't build a modern website. Other performances concerns might be connected with memory management or disk caches, but in general, we are safe to say that this web server cannot handle any production load (i.e. multiple users accessing the web site at the same time and expecting good quality of service).</p>
<p>This is hardly surprising. After all, we didn't want to deal with TCP/IP connections to focus on our business, so we delegated this to other coders who maintain the framework. The framework's authors, in turn, want to focus on things like middleware, routes, proper handling of HTTP methods, and so on. They don't want to spend time trying to optimise the performances of the "multi-user" experience. This is especially true in the Python world (and somehow less true for Node.js, for example): Python is not heavily concurrency-oriented, and both the style of programming and the performances are not favouring fast, non-blocking applications. This is changing lately, with async and improvements in the interpreter, but I leave this for another post.</p>
<p>So, now that we have a full-fledged HTTP service, we need to make it so fast that users won't even notice this is not running locally on their computer.</p>
<h2 id="3-concurrency-and-facades">3 Concurrency and façades<a class="headerlink" href="#3-concurrency-and-facades" title="Permanent link">¶</a></h2>
<h3 id="31-rationale">3.1 Rationale<a class="headerlink" href="#31-rationale" title="Permanent link">¶</a></h3>
<p>Well, whenever you have performance issues, just go for concurrency. Now you have many problems!
(see <a href="https://twitter.com/davidlohr/status/288786300067270656?lang=en">here</a>)</p>
<p>Yes, concurrency solves many problems and it's the source of just as much, so we need to find a way to use it in the safest and less complicated way. We basically might want to add a layer that runs the framework in some concurrent way, without requiring us to change anything in the framework itself.</p>
<p>And whenever you have to homogenise different things just create a layer of indirection. This solves any problem but one. (see <a href="https://en.wikipedia.org/wiki/Fundamental_theorem_of_software_engineering">here</a>)</p>
<p>So we need to create a layer that runs our service in a concurrent way, but we also want to keep it detached from the specific implementation of the service, that is independent of the framework or library that we are using.</p>
<h3 id="32-implementation">3.2 Implementation<a class="headerlink" href="#32-implementation" title="Permanent link">¶</a></h3>
<p>In this case, the solution is that of giving a <em>specification</em> of the API that web frameworks have to expose, in order to be usable by independent third-party components. In the Python world, this set of rules has been named WSGI, the Web Server Gateway Interface, but such interfaces exist for other languages such as Java or Ruby. The "gateway" mentioned here is the part of the system outside the framework, which in this discussion is the part that deals with production performances. Through WSGI we are defining a way for frameworks to expose a common interface, leaving people interested in concurrency free to implement something independently.</p>
<p>If the framework is compatible with the gateway interface, we can add software that deals with concurrency and uses the framework through the compatibility layer. Such a component is a production-ready HTTP server, and two common choices in the Python world are Gunicorn and uWSGI.</p>
<p>Production-ready HTTP server means that the software understands HTTP as the development server already did, but at the same time pushes performances in order to sustain a bigger workload, and as we said before this is done through concurrency.</p>
<p>Flask is compatible with WSGI, so we can make it work with Gunicorn. To install it in our virtual environment run <code>pip install gunicorn</code> and set it up creating a file names <code>wsgi.py</code> with the following content</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">app</span> <span class="kn">import</span> <span class="n">application</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">application</span><span class="o">.</span><span class="n">run</span><span class="p">()</span>
</code></pre></div>
<p>To run Gunicorn specify the number of concurrent instances and the external port</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>gunicorn<span class="w"> </span>--workers<span class="w"> </span><span class="m">3</span><span class="w"> </span>--bind<span class="w"> </span><span class="m">0</span>.0.0.0:8000<span class="w"> </span>wsgi
<span class="o">[</span><span class="m">2020</span>-02-12<span class="w"> </span><span class="m">18</span>:39:07<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="m">13393</span><span class="o">]</span><span class="w"> </span><span class="o">[</span>INFO<span class="o">]</span><span class="w"> </span>Starting<span class="w"> </span>gunicorn<span class="w"> </span><span class="m">20</span>.0.4
<span class="o">[</span><span class="m">2020</span>-02-12<span class="w"> </span><span class="m">18</span>:39:07<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="m">13393</span><span class="o">]</span><span class="w"> </span><span class="o">[</span>INFO<span class="o">]</span><span class="w"> </span>Listening<span class="w"> </span>at:<span class="w"> </span>http://0.0.0.0:8000<span class="w"> </span><span class="o">(</span><span class="m">13393</span><span class="o">)</span>
<span class="o">[</span><span class="m">2020</span>-02-12<span class="w"> </span><span class="m">18</span>:39:07<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="m">13393</span><span class="o">]</span><span class="w"> </span><span class="o">[</span>INFO<span class="o">]</span><span class="w"> </span>Using<span class="w"> </span>worker:<span class="w"> </span>sync
<span class="o">[</span><span class="m">2020</span>-02-12<span class="w"> </span><span class="m">18</span>:39:07<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="m">13396</span><span class="o">]</span><span class="w"> </span><span class="o">[</span>INFO<span class="o">]</span><span class="w"> </span>Booting<span class="w"> </span>worker<span class="w"> </span>with<span class="w"> </span>pid:<span class="w"> </span><span class="m">13396</span>
<span class="o">[</span><span class="m">2020</span>-02-12<span class="w"> </span><span class="m">18</span>:39:07<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="m">13397</span><span class="o">]</span><span class="w"> </span><span class="o">[</span>INFO<span class="o">]</span><span class="w"> </span>Booting<span class="w"> </span>worker<span class="w"> </span>with<span class="w"> </span>pid:<span class="w"> </span><span class="m">13397</span>
<span class="o">[</span><span class="m">2020</span>-02-12<span class="w"> </span><span class="m">18</span>:39:07<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="m">13398</span><span class="o">]</span><span class="w"> </span><span class="o">[</span>INFO<span class="o">]</span><span class="w"> </span>Booting<span class="w"> </span>worker<span class="w"> </span>with<span class="w"> </span>pid:<span class="w"> </span><span class="m">13398</span>
</code></pre></div>
<p>As you can see, Gunicorn has the concept of <em>workers</em> which are a generic way to express concurrency. Specifically, Gunicorn implements a pre-fork worker model, which means that it (pre)creates a different Unix process for each worker. You can check this running <code>ps</code></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ps<span class="w"> </span>ax<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>gunicorn
<span class="m">14919</span><span class="w"> </span>pts/1<span class="w"> </span>S+<span class="w"> </span><span class="m">0</span>:00<span class="w"> </span>~/venv3/bin/python3<span class="w"> </span>~/venv3/bin/gunicorn<span class="w"> </span>--workers<span class="w"> </span><span class="m">3</span><span class="w"> </span>--bind<span class="w"> </span><span class="m">0</span>.0.0.0:8000<span class="w"> </span>wsgi
<span class="m">14922</span><span class="w"> </span>pts/1<span class="w"> </span>S+<span class="w"> </span><span class="m">0</span>:00<span class="w"> </span>~/venv3/bin/python3<span class="w"> </span>~/venv3/bin/gunicorn<span class="w"> </span>--workers<span class="w"> </span><span class="m">3</span><span class="w"> </span>--bind<span class="w"> </span><span class="m">0</span>.0.0.0:8000<span class="w"> </span>wsgi
<span class="m">14923</span><span class="w"> </span>pts/1<span class="w"> </span>S+<span class="w"> </span><span class="m">0</span>:00<span class="w"> </span>~/venv3/bin/python3<span class="w"> </span>~/venv3/bin/gunicorn<span class="w"> </span>--workers<span class="w"> </span><span class="m">3</span><span class="w"> </span>--bind<span class="w"> </span><span class="m">0</span>.0.0.0:8000<span class="w"> </span>wsgi
<span class="m">14924</span><span class="w"> </span>pts/1<span class="w"> </span>S+<span class="w"> </span><span class="m">0</span>:00<span class="w"> </span>~/venv3/bin/python3<span class="w"> </span>~/venv3/bin/gunicorn<span class="w"> </span>--workers<span class="w"> </span><span class="m">3</span><span class="w"> </span>--bind<span class="w"> </span><span class="m">0</span>.0.0.0:8000<span class="w"> </span>wsgi
</code></pre></div>
<p>Using processes is just one of the two ways to implement concurrency in a Unix system, the other being using threads. The benefits and demerits of each solution are outside the scope of this post, however. For the time being just remember that you are dealing with multiple workers that process incoming requests asynchronously, thus implementing a non-blocking server, ready to accept multiple connections.</p>
<h3 id="33-resources">3.3 Resources<a class="headerlink" href="#33-resources" title="Permanent link">¶</a></h3>
<p>These resources provide more detailed information on the topics discussed in this section</p>
<ul>
<li>The <a href="https://wsgi.readthedocs.io/en/latest/index.html">WSGI official documentation</a> and the <a href="https://en.wikipedia.org/wiki/Web_Server_Gateway_Interface">Wikipedia page
</a></li>
<li>The homepages of <a href="https://gunicorn.org/">Gunicorn</a> and <a href="https://uwsgi-docs.readthedocs.io/en/latest/">uWSGI</a></li>
<li>A good entry point for your journey into the crazy world of concurrency: <a href="https://en.wikipedia.org/wiki/Multithreading_(computer_architecture)">multithreading</a>.</li>
<li>The source code of this example is available <a href="https://github.com/lgiordani/dissecting-a-web-stack-code/tree/master/3_concurrency_and_facades">here</a></li>
</ul>
<h3 id="34-issues">3.4 Issues<a class="headerlink" href="#34-issues" title="Permanent link">¶</a></h3>
<p>Using a Gunicorn we have now a production-ready HTTP server, and apparently implemented everything we need. There are still many considerations and missing pieces, though.</p>
<h4 id="performances-again">Performances (again)<a class="headerlink" href="#performances-again" title="Permanent link">¶</a></h4>
<p>Are 3 workers enough to sustain the load of our new killer mobile application? We expect thousands of visitors per minute, so maybe we should add some. But while we increase the amount of workers, we have to keep in mind that the machine we are using has a finite amount of CPU power and memory. So, once again, we have to focus on performances, and in particular on scalability: how can we keep adding workers without having to stop the application, replace the machine with a more powerful one, and restart the service?</p>
<h4 id="embrace-change">Embrace change<a class="headerlink" href="#embrace-change" title="Permanent link">¶</a></h4>
<p>This is not the only problem we have to face in production. An important aspect of technology is that it changes over time, as new and (hopefully) better solutions become widespread. We usually design systems dividing them as much as possible into communicating layers exactly because we want to be free to replace a layer with something else, be it a simpler component or a more advanced one, one with better performances or maybe just a cheaper one. So, once again, we want to be able to evolve the underlying system keeping the same interface, exactly as we did in the case of web frameworks.</p>
<h4 id="https_1">HTTPS<a class="headerlink" href="#https_1" title="Permanent link">¶</a></h4>
<p>Another missing part of the system is HTTPS. Gunicorn and uWSGI do not understand the HTTPS protocol, so we need something in front of them that will deal with the "S" part of the protocol, leaving the "HTTP" part to the internal layers.</p>
<h4 id="load-balancers">Load balancers<a class="headerlink" href="#load-balancers" title="Permanent link">¶</a></h4>
<p>In general, a <em>load balancer</em> is just a component in a system that distributes work among a pool of workers. Gunicorn is already distributing load among its workers, so this is not a new concept, but we generally want to do it on a bigger level, among machines or among entire systems. Load balancing can be hierarchical and be structured on many levels. We can also assign more importance to some components of the system, flagging them as ready to accept more load (for example because their hardware is better). Load balancers are extremely important in network services, and the definition of load can be extremely different from system to system: generally speaking, in a Web service the number of connections is the standard measure of the load, as we assume that on average all connections bring the same amount of work to the system.</p>
<h4 id="reverse-proxies">Reverse proxies<a class="headerlink" href="#reverse-proxies" title="Permanent link">¶</a></h4>
<p>Load balancers are forward proxies, as they allow a client to contact any server in a pool. At the same time, a <em>reverse proxy</em> allows a client to retrieve data produced by several systems through the same entry point. Reverse proxies are a perfect way to route HTTP requests to sub-systems that can be implemented with different technologies. For example, you might want to have part of the system implemented with Python, using Django and Postgres, and another part served by an AWS Lambda function written in Go and connected with a non-relational database such as DynamoDB. Usually, in HTTP services this choice is made according to the URL (for example routing every URL that begins with <code>/api/</code>).</p>
<h4 id="logic">Logic<a class="headerlink" href="#logic" title="Permanent link">¶</a></h4>
<p>We also want a layer that can implement a certain amount of logic, to manage simple rules that are not related to the service we implemented. A typical example is that of HTTP redirections: what happens if a user accesses the service with an <code>http://</code> prefix instead of <code>https://</code>? The correct way to deal with this is through an HTTP 301 code, but you don't want such a request to reach your framework, wasting resources for such a simple task.</p>
<h2 id="4-the-web-server">4 The Web server<a class="headerlink" href="#4-the-web-server" title="Permanent link">¶</a></h2>
<h3 id="41-rationale">4.1 Rationale<a class="headerlink" href="#41-rationale" title="Permanent link">¶</a></h3>
<p>The general label of <em>Web server</em> is given to software that performs the tasks we discussed. Two very common choices for this part of the system are nginx and Apache, two open source projects that are currently leading the market. With different technical approaches, they both implement all the features we discussed in the previous section (and many more).</p>
<h3 id="42-implementation">4.2 Implementation<a class="headerlink" href="#42-implementation" title="Permanent link">¶</a></h3>
<p>To test nginx without having to fight with the OS and install too many packages we can use Docker. Docker is useful to simulate a multi-machine environment, but it might also be your technology of choice for the actual production environment (AWS ECS works with Docker containers, for example).</p>
<p>The base configuration that we will run is very simple. One container will contain the Flask code and run the framework with Gunicorn, while the other container will run nginx. Gunicorn will serve HTTP on the internal port 8000, not exposed by Docker and thus not reachable from our browser, while nignx will expose port 80, the traditional HTTP port.</p>
<p>In the same directory of the file <code>wsgi.py</code>, create a <code>Dockerfile</code></p>
<div class="highlight"><pre><span></span><code><span class="k">FROM</span><span class="w"> </span><span class="s">python:3.6</span>
<span class="k">ADD</span><span class="w"> </span>app<span class="w"> </span>/app
<span class="k">ADD</span><span class="w"> </span>wsgi.py<span class="w"> </span>/
<span class="k">WORKDIR</span><span class="w"> </span><span class="s">.</span>
<span class="k">RUN</span><span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>flask<span class="w"> </span>gunicorn
<span class="k">EXPOSE</span><span class="w"> </span><span class="s">8000</span>
</code></pre></div>
<p>This starts from a Python Docker image, adds the <code>app</code> directory and the <code>wsgi.py</code> file, and installs Gunicorn. Now create a configuration for nginx in a file called <code>nginx.conf</code> in the same directory</p>
<div class="highlight"><pre><span></span><code><span class="k">server</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kn">listen</span><span class="w"> </span><span class="mi">80</span><span class="p">;</span>
<span class="w"> </span><span class="kn">server_name</span><span class="w"> </span><span class="s">localhost</span><span class="p">;</span>
<span class="w"> </span><span class="kn">location</span><span class="w"> </span><span class="s">/</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kn">proxy_pass</span><span class="w"> </span><span class="s">http://application:8000/</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>This defines a server that listens on port 80 and that connects all the URL starting with <code>/</code> with a server called <code>application</code> on port 8000, which is the container running Gunicorn.</p>
<p>Last, create a file <code>docker-compose.yml</code> that will describe the configuration of the containers.</p>
<div class="highlight"><pre><span></span><code><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="s">"3.7"</span>
<span class="nt">services</span><span class="p">:</span>
<span class="w"> </span><span class="nt">application</span><span class="p">:</span>
<span class="w"> </span><span class="nt">build</span><span class="p">:</span>
<span class="w"> </span><span class="nt">context</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">.</span>
<span class="w"> </span><span class="nt">dockerfile</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Dockerfile</span>
<span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">gunicorn --workers 3 --bind 0.0.0.0:8000 wsgi</span>
<span class="w"> </span><span class="nt">expose</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">8000</span>
<span class="w"> </span><span class="nt">nginx</span><span class="p">:</span>
<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">nginx</span>
<span class="w"> </span><span class="nt">volumes</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">./nginx.conf:/etc/nginx/conf.d/default.conf</span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">8080:80</span>
<span class="w"> </span><span class="nt">depends_on</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">application</span>
</code></pre></div>
<p>As you can see the name <code>application</code> that we mentioned in the nginx configuration file is not a magic string, but is the name we assigned to the Gunicorn container in the Docker Compose configuration. Please note that nginx listens on port 80 inside the container, but the port is published as 8080 on the host.</p>
<p>To create this infrastructure we need to install Docker Compose in our virtual environment through <code>pip install docker-compose</code>. I also created a file named <code>.env</code> with the name of the project</p>
<div class="highlight"><pre><span></span><code><span class="nv">COMPOSE_PROJECT_NAME</span><span class="o">=</span>service
</code></pre></div>
<p>At this point you can run Docker Compose with <code>docker-compose up -d</code></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker-compose<span class="w"> </span>up<span class="w"> </span>-d
Creating<span class="w"> </span>network<span class="w"> </span><span class="s2">"service_default"</span><span class="w"> </span>with<span class="w"> </span>the<span class="w"> </span>default<span class="w"> </span>driver
Creating<span class="w"> </span>service_application_1<span class="w"> </span>...<span class="w"> </span><span class="k">done</span>
Creating<span class="w"> </span>service_nginx_1<span class="w"> </span>...<span class="w"> </span><span class="k">done</span>
</code></pre></div>
<p>If everything is working correctly, opening the browser and visiting <code>localhost:8080</code> should show you the HTML page Flask is serving.</p>
<p>Through <code>docker-compose logs</code> we can check what services are doing. We can recognise the output of Gunicorn in the logs of the service named <code>application</code></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker-compose<span class="w"> </span>logs<span class="w"> </span>application
Attaching<span class="w"> </span>to<span class="w"> </span>service_application_1
application_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">2020</span>-02-14<span class="w"> </span><span class="m">08</span>:35:42<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="m">1</span><span class="o">]</span><span class="w"> </span><span class="o">[</span>INFO<span class="o">]</span><span class="w"> </span>Starting<span class="w"> </span>gunicorn<span class="w"> </span><span class="m">20</span>.0.4
application_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">2020</span>-02-14<span class="w"> </span><span class="m">08</span>:35:42<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="m">1</span><span class="o">]</span><span class="w"> </span><span class="o">[</span>INFO<span class="o">]</span><span class="w"> </span>Listening<span class="w"> </span>at:<span class="w"> </span>http://0.0.0.0:8000<span class="w"> </span><span class="o">(</span><span class="m">1</span><span class="o">)</span>
application_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">2020</span>-02-14<span class="w"> </span><span class="m">08</span>:35:42<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="m">1</span><span class="o">]</span><span class="w"> </span><span class="o">[</span>INFO<span class="o">]</span><span class="w"> </span>Using<span class="w"> </span>worker:<span class="w"> </span>sync
application_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">2020</span>-02-14<span class="w"> </span><span class="m">08</span>:35:42<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="m">8</span><span class="o">]</span><span class="w"> </span><span class="o">[</span>INFO<span class="o">]</span><span class="w"> </span>Booting<span class="w"> </span>worker<span class="w"> </span>with<span class="w"> </span>pid:<span class="w"> </span><span class="m">8</span>
application_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">2020</span>-02-14<span class="w"> </span><span class="m">08</span>:35:42<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="m">9</span><span class="o">]</span><span class="w"> </span><span class="o">[</span>INFO<span class="o">]</span><span class="w"> </span>Booting<span class="w"> </span>worker<span class="w"> </span>with<span class="w"> </span>pid:<span class="w"> </span><span class="m">9</span>
application_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">2020</span>-02-14<span class="w"> </span><span class="m">08</span>:35:42<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="m">10</span><span class="o">]</span><span class="w"> </span><span class="o">[</span>INFO<span class="o">]</span><span class="w"> </span>Booting<span class="w"> </span>worker<span class="w"> </span>with<span class="w"> </span>pid:<span class="w"> </span><span class="m">10</span>
</code></pre></div>
<p>but the one we are mostly interested with now is the service named <code>nginx</code>, so let's follow the logs in real-time with <code>docker-compose logs -f nginx</code>. Refresh the <code>localhost</code> page you visited with the browser, and the container should output something like</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker-compose<span class="w"> </span>logs<span class="w"> </span>-f<span class="w"> </span>nginx
Attaching<span class="w"> </span>to<span class="w"> </span>service_nginx_1
nginx_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="m">192</span>.168.192.1<span class="w"> </span>-<span class="w"> </span>-<span class="w"> </span><span class="o">[</span><span class="m">14</span>/Feb/2020:08:42:20<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span><span class="s2">"GET / HTTP/1.1"</span><span class="w"> </span><span class="m">200</span><span class="w"> </span><span class="m">13</span><span class="w"> </span><span class="s2">"-"</span><span class="w"> </span><span class="s2">"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0"</span><span class="w"> </span><span class="s2">"-"</span>
</code></pre></div>
<p>which is the standard log format of nginx. It shows the IP address of the client (<code>192.168.192.1</code>), the connection timestamp, the HTTP request and the response status code (200), plus other information on the client itself.</p>
<p>Let's now increase the number of services, to see the load balancing mechanism in action. To do this, first we need to change the log format of nginx to show the IP address of the machine that served the request. Change the <code>nginx.conf</code> file adding the <code>log_format</code> and <code>access_log</code> options</p>
<div class="highlight"><pre><span></span><code><span class="k">log_format</span><span class="w"> </span><span class="s">upstreamlog</span><span class="w"> </span><span class="s">'[</span><span class="nv">$time_local]</span><span class="w"> </span><span class="nv">$host</span><span class="w"> </span><span class="s">to:</span><span class="w"> </span><span class="nv">$upstream_addr:</span><span class="w"> </span><span class="nv">$request</span><span class="w"> </span><span class="nv">$status'</span><span class="p">;</span>
<span class="k">server</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kn">listen</span><span class="w"> </span><span class="mi">80</span><span class="p">;</span>
<span class="w"> </span><span class="kn">server_name</span><span class="w"> </span><span class="s">localhost</span><span class="p">;</span>
<span class="w"> </span><span class="kn">location</span><span class="w"> </span><span class="s">/</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kn">proxy_pass</span><span class="w"> </span><span class="s">http://application:8000</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="kn">access_log</span><span class="w"> </span><span class="s">/var/log/nginx/access.log</span><span class="w"> </span><span class="s">upstreamlog</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>The <code>$upstream_addr</code> variable is the one that contains the IP address of the server proxied by nginx. Now run <code>docker-compose down</code> to stop all containers and then <code>docker-compose up -d --scale application=3</code> to start them again</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker-compose<span class="w"> </span>down
Stopping<span class="w"> </span>service_nginx_1<span class="w"> </span>...<span class="w"> </span><span class="k">done</span>
Stopping<span class="w"> </span>service_application_1<span class="w"> </span>...<span class="w"> </span><span class="k">done</span>
Removing<span class="w"> </span>service_nginx_1<span class="w"> </span>...<span class="w"> </span><span class="k">done</span>
Removing<span class="w"> </span>service_application_1<span class="w"> </span>...<span class="w"> </span><span class="k">done</span>
Removing<span class="w"> </span>network<span class="w"> </span>service_default
$<span class="w"> </span>docker-compose<span class="w"> </span>up<span class="w"> </span>-d<span class="w"> </span>--scale<span class="w"> </span><span class="nv">application</span><span class="o">=</span><span class="m">3</span>
Creating<span class="w"> </span>network<span class="w"> </span><span class="s2">"service_default"</span><span class="w"> </span>with<span class="w"> </span>the<span class="w"> </span>default<span class="w"> </span>driver
Creating<span class="w"> </span>service_application_1<span class="w"> </span>...<span class="w"> </span><span class="k">done</span>
Creating<span class="w"> </span>service_application_2<span class="w"> </span>...<span class="w"> </span><span class="k">done</span>
Creating<span class="w"> </span>service_application_3<span class="w"> </span>...<span class="w"> </span><span class="k">done</span>
Creating<span class="w"> </span>service_nginx_1<span class="w"> </span>...<span class="w"> </span><span class="k">done</span>
</code></pre></div>
<p>As you can see, Docker Compose runs now 3 containers for the <code>application</code> service. If you open the logs stream and visit the page in the browser you will now see a slightly different output</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker-compose<span class="w"> </span>logs<span class="w"> </span>-f<span class="w"> </span>nginx
Attaching<span class="w"> </span>to<span class="w"> </span>service_nginx_1
nginx_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">14</span>/Feb/2020:09:00:16<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span>localhost<span class="w"> </span>to:<span class="w"> </span><span class="m">192</span>.168.240.4:8000:<span class="w"> </span>GET<span class="w"> </span>/<span class="w"> </span>HTTP/1.1<span class="w"> </span><span class="m">200</span>
</code></pre></div>
<p>where you can spot <code>to: 192.168.240.4:8000</code> which is the IP address of one of the application containers. Please note that the IP address you see might be different, as it depends on the Docker network settings. If you now visit the page again multiple times you should notice a change in the upstream address, something like</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker-compose<span class="w"> </span>logs<span class="w"> </span>-f<span class="w"> </span>nginx
Attaching<span class="w"> </span>to<span class="w"> </span>service_nginx_1
nginx_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">14</span>/Feb/2020:09:00:16<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span>localhost<span class="w"> </span>to:<span class="w"> </span><span class="m">192</span>.168.240.4:8000:<span class="w"> </span>GET<span class="w"> </span>/<span class="w"> </span>HTTP/1.1<span class="w"> </span><span class="m">200</span>
nginx_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">14</span>/Feb/2020:09:00:17<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span>localhost<span class="w"> </span>to:<span class="w"> </span><span class="m">192</span>.168.240.2:8000:<span class="w"> </span>GET<span class="w"> </span>/<span class="w"> </span>HTTP/1.1<span class="w"> </span><span class="m">200</span>
nginx_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">14</span>/Feb/2020:09:00:17<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span>localhost<span class="w"> </span>to:<span class="w"> </span><span class="m">192</span>.168.240.3:8000:<span class="w"> </span>GET<span class="w"> </span>/<span class="w"> </span>HTTP/1.1<span class="w"> </span><span class="m">200</span>
nginx_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">14</span>/Feb/2020:09:00:17<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span>localhost<span class="w"> </span>to:<span class="w"> </span><span class="m">192</span>.168.240.4:8000:<span class="w"> </span>GET<span class="w"> </span>/<span class="w"> </span>HTTP/1.1<span class="w"> </span><span class="m">200</span>
nginx_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">14</span>/Feb/2020:09:00:17<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span>localhost<span class="w"> </span>to:<span class="w"> </span><span class="m">192</span>.168.240.2:8000:<span class="w"> </span>GET<span class="w"> </span>/<span class="w"> </span>HTTP/1.1<span class="w"> </span><span class="m">200</span>
</code></pre></div>
<p>This shows that nginx is performing load balancing, but to tell the truth this is happening through Docker's DNS, and not by an explicit action performed by the web server. We can verify this accessing the nginx container and running <code>dig application</code> (you need to run <code>apt update</code> and <code>apt install dnsutils</code> to install <code>dig</code>)</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker-compose<span class="w"> </span><span class="nb">exec</span><span class="w"> </span>nginx<span class="w"> </span>/bin/bash
root@99c2f348140e:/#<span class="w"> </span>apt<span class="w"> </span>update
root@99c2f348140e:/#<span class="w"> </span>apt<span class="w"> </span>install<span class="w"> </span>-y<span class="w"> </span>dnsutils
root@99c2f348140e:/#<span class="w"> </span>dig<span class="w"> </span>application
<span class="p">;</span><span class="w"> </span><<>><span class="w"> </span>DiG<span class="w"> </span><span class="m">9</span>.11.5-P4-5.1-Debian<span class="w"> </span><<>><span class="w"> </span>application
<span class="p">;;</span><span class="w"> </span>global<span class="w"> </span>options:<span class="w"> </span>+cmd
<span class="p">;;</span><span class="w"> </span>Got<span class="w"> </span>answer:
<span class="p">;;</span><span class="w"> </span>->>HEADER<span class="s"><<- opco</span>de:<span class="w"> </span>QUERY,<span class="w"> </span>status:<span class="w"> </span>NOERROR,<span class="w"> </span>id:<span class="w"> </span><span class="m">7221</span>
<span class="p">;;</span><span class="w"> </span>flags:<span class="w"> </span>qr<span class="w"> </span>rd<span class="w"> </span>ra<span class="p">;</span><span class="w"> </span>QUERY:<span class="w"> </span><span class="m">1</span>,<span class="w"> </span>ANSWER:<span class="w"> </span><span class="m">3</span>,<span class="w"> </span>AUTHORITY:<span class="w"> </span><span class="m">0</span>,<span class="w"> </span>ADDITIONAL:<span class="w"> </span><span class="m">0</span>
<span class="p">;;</span><span class="w"> </span>QUESTION<span class="w"> </span>SECTION:
<span class="p">;</span>application.<span class="w"> </span>IN<span class="w"> </span>A
<span class="p">;;</span><span class="w"> </span>ANSWER<span class="w"> </span>SECTION:
application.<span class="w"> </span><span class="m">600</span><span class="w"> </span>IN<span class="w"> </span>A<span class="w"> </span><span class="m">192</span>.168.240.2
application.<span class="w"> </span><span class="m">600</span><span class="w"> </span>IN<span class="w"> </span>A<span class="w"> </span><span class="m">192</span>.168.240.4
application.<span class="w"> </span><span class="m">600</span><span class="w"> </span>IN<span class="w"> </span>A<span class="w"> </span><span class="m">192</span>.168.240.3
<span class="p">;;</span><span class="w"> </span>Query<span class="w"> </span>time:<span class="w"> </span><span class="m">1</span><span class="w"> </span>msec
<span class="p">;;</span><span class="w"> </span>SERVER:<span class="w"> </span><span class="m">127</span>.0.0.11#53<span class="o">(</span><span class="m">127</span>.0.0.11<span class="o">)</span>
<span class="p">;;</span><span class="w"> </span>WHEN:<span class="w"> </span>Fri<span class="w"> </span>Feb<span class="w"> </span><span class="m">14</span><span class="w"> </span><span class="m">09</span>:57:24<span class="w"> </span>UTC<span class="w"> </span><span class="m">2020</span>
<span class="p">;;</span><span class="w"> </span>MSG<span class="w"> </span>SIZE<span class="w"> </span>rcvd:<span class="w"> </span><span class="m">110</span>
</code></pre></div>
<p>To see load balancing performed by nginx we can explicitly define two services and assign them different weights. Run <code>docker-compose down</code> and change the nginx configuration to</p>
<div class="highlight"><pre><span></span><code><span class="k">upstream</span><span class="w"> </span><span class="s">app</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kn">server</span><span class="w"> </span><span class="n">application1</span><span class="p">:</span><span class="mi">8000</span><span class="w"> </span><span class="s">weight=3</span><span class="p">;</span>
<span class="w"> </span><span class="kn">server</span><span class="w"> </span><span class="n">application2</span><span class="p">:</span><span class="mi">8000</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">log_format</span><span class="w"> </span><span class="s">upstreamlog</span><span class="w"> </span><span class="s">'[</span><span class="nv">$time_local]</span><span class="w"> </span><span class="nv">$host</span><span class="w"> </span><span class="s">to:</span><span class="w"> </span><span class="nv">$upstream_addr:</span><span class="w"> </span><span class="nv">$request</span><span class="w"> </span><span class="nv">$status'</span><span class="p">;</span>
<span class="k">server</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kn">listen</span><span class="w"> </span><span class="mi">80</span><span class="p">;</span>
<span class="w"> </span><span class="kn">server_name</span><span class="w"> </span><span class="s">localhost</span><span class="p">;</span>
<span class="w"> </span><span class="kn">location</span><span class="w"> </span><span class="s">/</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kn">proxy_pass</span><span class="w"> </span><span class="s">http://app</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="kn">access_log</span><span class="w"> </span><span class="s">/var/log/nginx/access.log</span><span class="w"> </span><span class="s">upstreamlog</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>We defined here an <code>upstream</code> structure that lists two different services, <code>application1</code> and <code>application2</code>, giving to the first one a weight of 3. This mean that each 4 requests, 3 will be routed to the first service, and one to the second service. Now nginx is not just relying on the DNS, but consciously choosing between two different services.</p>
<p>Let's define the services accordingly in the Docker Compose configuration file</p>
<div class="highlight"><pre><span></span><code>version: "3"
services:
application1:
build:
context: .
dockerfile: Dockerfile
command: gunicorn --workers 6 --bind 0.0.0.0:8000 wsgi
expose:
<span class="k">-</span> 8000
application2:
build:
context: .
dockerfile: Dockerfile
command: gunicorn --workers 3 --bind 0.0.0.0:8000 wsgi
expose:
<span class="k">-</span> 8000
nginx:
image: nginx
volumes:
<span class="k">-</span> ./nginx.conf:/etc/nginx/conf.d/default.conf
ports:
<span class="k">-</span> 80:80
depends_on:
<span class="k">-</span> application1
<span class="k">-</span> application2
</code></pre></div>
<p>I basically duplicated the definition of <code>application</code>, but the first service is running now 6 workers, just for the sake of showing a possible difference between the two. Now run <code>docker-compose up -d</code> and <code>docker-compose logs -f nginx</code>. If you refresh the page on the browser multiple times you will see something like</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker-compose<span class="w"> </span>logs<span class="w"> </span>-f<span class="w"> </span>nginx
Attaching<span class="w"> </span>to<span class="w"> </span>service_nginx_1
nginx_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">14</span>/Feb/2020:11:03:25<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span>localhost<span class="w"> </span>to:<span class="w"> </span><span class="m">172</span>.18.0.2:8000:<span class="w"> </span>GET<span class="w"> </span>/<span class="w"> </span>HTTP/1.1<span class="w"> </span><span class="m">200</span>
nginx_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">14</span>/Feb/2020:11:03:25<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span>localhost<span class="w"> </span>to:<span class="w"> </span><span class="m">172</span>.18.0.2:8000:<span class="w"> </span>GET<span class="w"> </span>/favicon.ico<span class="w"> </span>HTTP/1.1<span class="w"> </span><span class="m">404</span>
nginx_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">14</span>/Feb/2020:11:03:30<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span>localhost<span class="w"> </span>to:<span class="w"> </span><span class="m">172</span>.18.0.3:8000:<span class="w"> </span>GET<span class="w"> </span>/<span class="w"> </span>HTTP/1.1<span class="w"> </span><span class="m">200</span>
nginx_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">14</span>/Feb/2020:11:03:31<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span>localhost<span class="w"> </span>to:<span class="w"> </span><span class="m">172</span>.18.0.2:8000:<span class="w"> </span>GET<span class="w"> </span>/<span class="w"> </span>HTTP/1.1<span class="w"> </span><span class="m">200</span>
nginx_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">14</span>/Feb/2020:11:03:32<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span>localhost<span class="w"> </span>to:<span class="w"> </span><span class="m">172</span>.18.0.2:8000:<span class="w"> </span>GET<span class="w"> </span>/<span class="w"> </span>HTTP/1.1<span class="w"> </span><span class="m">200</span>
nginx_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">14</span>/Feb/2020:11:03:33<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span>localhost<span class="w"> </span>to:<span class="w"> </span><span class="m">172</span>.18.0.2:8000:<span class="w"> </span>GET<span class="w"> </span>/<span class="w"> </span>HTTP/1.1<span class="w"> </span><span class="m">200</span>
nginx_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">14</span>/Feb/2020:11:03:33<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span>localhost<span class="w"> </span>to:<span class="w"> </span><span class="m">172</span>.18.0.3:8000:<span class="w"> </span>GET<span class="w"> </span>/<span class="w"> </span>HTTP/1.1<span class="w"> </span><span class="m">200</span>
nginx_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">14</span>/Feb/2020:11:03:34<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span>localhost<span class="w"> </span>to:<span class="w"> </span><span class="m">172</span>.18.0.2:8000:<span class="w"> </span>GET<span class="w"> </span>/<span class="w"> </span>HTTP/1.1<span class="w"> </span><span class="m">200</span>
nginx_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">14</span>/Feb/2020:11:03:34<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span>localhost<span class="w"> </span>to:<span class="w"> </span><span class="m">172</span>.18.0.2:8000:<span class="w"> </span>GET<span class="w"> </span>/<span class="w"> </span>HTTP/1.1<span class="w"> </span><span class="m">200</span>
nginx_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">14</span>/Feb/2020:11:03:35<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span>localhost<span class="w"> </span>to:<span class="w"> </span><span class="m">172</span>.18.0.2:8000:<span class="w"> </span>GET<span class="w"> </span>/<span class="w"> </span>HTTP/1.1<span class="w"> </span><span class="m">200</span>
nginx_1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="o">[</span><span class="m">14</span>/Feb/2020:11:03:35<span class="w"> </span>+0000<span class="o">]</span><span class="w"> </span>localhost<span class="w"> </span>to:<span class="w"> </span><span class="m">172</span>.18.0.3:8000:<span class="w"> </span>GET<span class="w"> </span>/<span class="w"> </span>HTTP/1.1<span class="w"> </span><span class="m">200</span>
</code></pre></div>
<p>where you can clearly notice the load balancing between <code>172.18.0.2</code> (<code>application1</code>) and <code>172.18.0.3</code> (<code>application2</code>) in action.</p>
<p>I will not show here an example of reverse proxy or HTTPS to prevent this post to become too long. You can find resources on those topics in the next section.</p>
<h3 id="43-resources">4.3 Resources<a class="headerlink" href="#43-resources" title="Permanent link">¶</a></h3>
<p>These resources provide more detailed information on the topics discussed in this section</p>
<ul>
<li>Docker Compose <a href="https://docs.docker.com/compose/">official documentation</a></li>
<li>nginx <a href="http://nginx.org/en/docs/">documentation</a>: in particular the sections about <a href="http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format">log_format</a> and <a href="http://nginx.org/en/docs/http/ngx_http_upstream_module.html#upstream">upstream</a> directives</li>
<li>How to <a href="https://docs.nginx.com/nginx/admin-guide/monitoring/logging/">configure logging</a> in nginx</li>
<li>How to <a href="https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/">configure load balancing</a> in nginx</li>
<li><a href="https://docs.nginx.com/nginx/admin-guide/security-controls/terminating-ssl-http/">Setting up an HTTPS Server</a> with nginx and <a href="https://www.humankode.com/ssl/create-a-selfsigned-certificate-for-nginx-in-5-minutes">how to created self-signed certificates</a></li>
<li>How to <a href="https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/">create a reverse proxy</a> with nginx, the documentation of the <a href="http://nginx.org/en/docs/http/ngx_http_core_module.html#location"><code>location</code></a> directive and <a href="https://www.digitalocean.com/community/tutorials/understanding-nginx-server-and-location-block-selection-algorithms">some insights</a> on the location choosing algorithms (one of the most complex parts of nginx)</li>
<li>The source code of this example is available <a href="https://github.com/lgiordani/dissecting-a-web-stack-code/tree/master/4_the_web_server">here</a></li>
</ul>
<h3 id="44-issues">4.4 Issues<a class="headerlink" href="#44-issues" title="Permanent link">¶</a></h3>
<p>Well, finally we can say that the job is done. Now we have a production-ready web server in front of our multi-threaded web framework and we can focus on writing Python code instead of dealing with HTTP headers.</p>
<p>Using a web server allows us to scale the infrastructure just adding new instances behind it, without interrupting the service. The HTTP concurrent server runs multiple instances of our framework, and the framework itself abstracts HTTP, mapping it to our high-level language.</p>
<h2 id="bonus-cloud-infrastructures">Bonus: cloud infrastructures<a class="headerlink" href="#bonus-cloud-infrastructures" title="Permanent link">¶</a></h2>
<p>Back in the early years of the Internet, companies used to have their own servers on-premise, and system administrators used to run the whole stack directly on the bare operating system. Needless to say, this was complicated, expensive, and failure-prone.</p>
<p>Nowadays "the cloud" is the way to go, so I want to briefly mention some components that can help you run such a web stack on AWS, which is the platform I know the most and the most widespread cloud provider in the world at the time of writing.</p>
<h3 id="elastic-beanstalk">Elastic Beanstalk<a class="headerlink" href="#elastic-beanstalk" title="Permanent link">¶</a></h3>
<p>This is the entry-level solution for simple applications, being a managed infrastructure that provides load balancing, auto-scaling, and monitoring. You can use several programming languages (among which Python and Node.js) and choose between different web servers like for example Apache or nginx. The components of an EB service are not hidden, but you don't have direct access to them, and you have to rely on configuration files to change the way they work. It's a good solution for simple services, but you will probably soon need more control.</p>
<p><a href="https://aws.amazon.com/elasticbeanstalk">Go to Elastic Beanstalk</a></p>
<h3 id="elastic-container-service-ecs">Elastic Container Service (ECS)<a class="headerlink" href="#elastic-container-service-ecs" title="Permanent link">¶</a></h3>
<p>With ECS you can run Docker containers grouping them in clusters and setting up auto-scale policies connected with metrics coming from CloudWatch. You have the choice of running them on EC2 instances (virtual machines) managed by you or on a serverless infrastructure called Fargate. ECS will run your Docker containers, but you still have to create DNS entries and load balancers on your own. You also have the choice of running your containers on Kubernetes using EKS (Elastic Kubernetes Service).</p>
<p><a href="https://aws.amazon.com/ecs/">Go to Elastic Container Service</a></p>
<h3 id="elastic-compute-cloud-ec2">Elastic Compute Cloud (EC2)<a class="headerlink" href="#elastic-compute-cloud-ec2" title="Permanent link">¶</a></h3>
<p>This is the bare metal of AWS, where you spin up stand-alone virtual machines or auto-scaling group of them. You can SSH into these instances and provide scripts to install and configure software. You can install here your application, web servers, databases, whatever you want. While this used to be the way to go at the very beginning of the cloud computing age I don't think you should go for it. There is so much a cloud provider can give you in terms of associated services like logs or monitoring, and in terms of performances, that it doesn't make sense to avoid using them. EC2 is still there, anyway, and if you run ECS on top of it you need to know what you can and what you can't do.</p>
<p><a href="https://aws.amazon.com/ec2/">Go to Elastic Compute Cloud</a></p>
<h3 id="elastic-load-balancing">Elastic Load Balancing<a class="headerlink" href="#elastic-load-balancing" title="Permanent link">¶</a></h3>
<p>While Network Load Balancers (NLB) manage pure TCP/IP connections, Application Load Balancers are dedicated to HTTP, and they can perform many of the services we need. They can reverse proxy through rules (that were recently improved) and they can terminate TLS, using certificates created in ACM (AWS Certificate Manager). As you can see, ALBs are a good replacement for a web server, even though they clearly lack the extreme configurability of a software. You can, however, use them as the first layer of load balancing, still using nginx or Apache behind them if you need some of the features they provide.</p>
<p><a href="https://aws.amazon.com/elasticloadbalancing/">Go to Elastic Load Balancing</a></p>
<h3 id="cloudfront">CloudFront<a class="headerlink" href="#cloudfront" title="Permanent link">¶</a></h3>
<p>CloudFront is a Content Delivery Network, that is a geographically-distributed cache that provides faster access to your content. While CDNs are not part of the stack that I discussed in this post I think it is worth mentioning CF as it can speed-up any static content, and also terminate TLS in connection with AWS Certificate Manager.</p>
<p><a href="https://aws.amazon.com/cloudfront/">Go to CloudFront</a></p>
<h2 id="conclusion">Conclusion<a class="headerlink" href="#conclusion" title="Permanent link">¶</a></h2>
<p>As you can see a web stack is a pretty rich set of components, and the reason behind them is often related to performances. There are a lot of technologies that we take for granted, and that fortunately have become easier to deploy, but I still believe a full-stack engineer should be aware not only of the existence of such layers, but also of their purpose and at least their basic configuration.</p>
<h2 id="feedback">Feedback<a class="headerlink" href="#feedback" title="Permanent link">¶</a></h2>
<p>Feel free to reach me on <a href="https://twitter.com/thedigicat">Twitter</a> if you have questions. The <a href="https://github.com/TheDigitalCatOnline/blog_source/issues">GitHub issues</a> page is the best place to submit corrections.</p>