The Digital Cat - Python2https://www.thedigitalcatonline.com/2022-09-18T23:00:00+00:00Adventures of a curious cat in the land of programmingFirst-class objects in Python - Higher-order functions, wrappers, and factories2021-03-09T16:00:00+00:002022-09-18T23:00:00+00:00Leonardo Giordanitag:www.thedigitalcatonline.com,2021-03-09:/blog/2021/03/09/first-class-objects-in-python/<p>My new book "First-class objects in Python" is out! Grab your <strong>FREE</strong> copy <a href="https://www.thedigitalcat.academy/freebie-first-class-objects">here</a>!</p><div class="imageblock"><img src="/images/first-class-objects-in-python.jpg"></div>Mau: a lightweight markup language2021-02-22T10:00:00+00:002021-02-25T18:00:00+00:00Leonardo Giordanitag:www.thedigitalcatonline.com,2021-02-22:/blog/2021/02/22/mau-a-lightweight-markup-language/<p>Mau is a lightweight markup language heavily inspired by AsciiDoc that makes is very easy to write blog posts or books.</p><h2 id="what-is-mau-fb3c">What is Mau?<a class="headerlink" href="#what-is-mau-fb3c" title="Permanent link">¶</a></h2><p>Mau is a lightweight markup language heavily inspired by AsciiDoc that makes is very easy to write blog posts or books.</p><p>The main goal of Mau is to provide a customisable markup language, reusing the good parts of AsciiDoc and providing a pure Python 3 implementation.</p><p>You can find Mau's source code on <a href="https://github.com/Project-Mau/mau">GitHub</a>.</p><h2 id="why-not-markdown-or-asciidoc-b535">Why not Markdown or AsciiDoc?<a class="headerlink" href="#why-not-markdown-or-asciidoc-b535" title="Permanent link">¶</a></h2><p>Markdown is a very good format, and I used it for all the posts in this blog so far. I grew increasingly unsatisfied, though, because of the lack of some features and the poor amount of customisation that it provides. When I wrote the second version of my book "Clean Architectures in Python" I considered using Markdown (through Pelican), but I couldn't find a good way to create tips and warnings. Recently, Python Markdown added a feature that allows to specify the file name for the source code, but the resulting HTML cannot easily be changed, making it difficult to achieve the graphical output I wanted.</p><p>AsciiDoc started as a Python project, but then was abandoned and eventually resurrected by Dan Allen with Asciidoctor. AsciiDoc has a lot of features and I consider it superior to Markdown, but Asciidoctor is a Ruby program, and this made it difficult for me to use it. In addition, the standard output of Asciidoctor is a nice single HTML page but again customising it is a pain. I had to struggle to add my Google Analytics code and a <code>sitemap.xml</code> to the book site.</p><p>I simply thought I could try to write my own tool, in a language that I know well (Python). It works, and I learned a lot writing it, so I'm definitely happy. I'd be delighted to know that this can be useful to other people, though.</p><h2 id="pelican-f581">Pelican<a class="headerlink" href="#pelican-f581" title="Permanent link">¶</a></h2><p>A reader for Mau source files is available in Pelican, you can find the code at <a href="https://github.com/getpelican/pelican-plugins/pull/1327">https://github.com/getpelican/pelican-plugins/pull/1327</a>. Simply add the code to your Pelican plugins directory and activate it adding <code>"mau_reader"</code> to <code>PLUGINS</code> in your file <code>pelicanconf.py</code>. The Mau reader processes only files with the <code>.mau</code> extension, so you can use Markdown/reStructuredText and Mau at the same time.</p><h2 id="development-f3c5">Development<a class="headerlink" href="#development-f3c5" title="Permanent link">¶</a></h2><p>If you are interested you can leave a star on the project on the <a href="https://github.com/Project-Mau/mau">GitHub page</a>, start using it, or contribute ideas, code, bugfixes.</p><h2 id="feedback-d845">Feedback<a class="headerlink" href="#feedback-d845" title="Permanent link">¶</a></h2><p>Feel free to reach me on <a href="https://twitter.com/thedigicat">Twitter</a> if you have questions. The <a href="https://github.com/TheDigitalCatOnline/blog_source/issues">GitHub issues</a> page is the best place to submit corrections.</p>The Digital Cat Youtube Channel2019-05-27T12:00:00+01:002019-05-27T12:00:00+01:00Leonardo Giordanitag:www.thedigitalcatonline.com,2019-05-27:/blog/2019/05/27/youtube-channel/<p>The new Youtube channel The Digital Cat is out in the wild</p><p>The Digital Cat <a href="https://www.youtube.com/channel/UCJ70w0WzWjWerpk3utRcAKA">Youtube channel</a> just launched!</p>
<p>The channel will host workshops and tutorial on Python and other languages, on operating systems, cryptography, and other topics that you can find here on this blog. I just finished recording the first part of my workshop "TDD in Python with pytest", which was successfully presented at PyCon UK, PyCon IT, PyCon Ireland, EuroPython and PyLadies London, and the 4 videos are already available on the channel.</p>
<div class="big-image">
<img src="/images/global/banner_large.jpg" alt="Youtube Channel banner" />
</div>
<p>You can subscribe to the channel at <a href="https://www.youtube.com/channel/UCJ70w0WzWjWerpk3utRcAKA">this link</a></p>
<p>I appreciate that many people struggle to follow a long post or a tutorial and prefer to see how someone works and hear a voice explaining the steps. Well, I'm not sure if they will like hearing my voice, but this channel is the attempts to reach that type of audience. These are my first videos, so the production is not at its best, but I hope I will improve it in the future.</p>
<p>I hope you will find them useful, feel free to post comments under the videos if you have suggestions, if you need clarifications, or just to say that you enjoyed them. Thanks!</p>Clean Architectures in Python: the book2018-12-20T08:00:00+01:002021-08-22T08:00:00+00:00Leonardo Giordanitag:www.thedigitalcatonline.com,2018-12-20:/blog/2018/12/20/cabook/<p>A practical approach to better software design</p><p>The second edition of the book "Clean Architectures in Python. A practical approach to better software design" is out!</p>
<div class="center-image">
<img src="/images/cabook/cover.jpg" alt="Cover" />
</div>
<p>You can read the book online on the new website <a href="https://www.thedigitalcatbooks.com/">The Digital Cat Books</a> or download a PDF version <a href="https://leanpub.com/clean-architectures-in-python">from Leanpub</a>. If you enjoy it, please tweet about it with the <code>#pycabook</code> hashtag.</p>
<p>So far more than 16,000 readers downloaded the book. Thank you all!</p>Useful pytest command line options2018-07-05T11:00:00+01:002018-07-05T11:00:00+01:00Leonardo Giordanitag:www.thedigitalcatonline.com,2018-07-05:/blog/2018/07/05/useful-pytest-command-line-options/<p>A curated list of useful command line options of the Python unit testing framework pytest</p><p>I recently gave a workshop on "TDD in Python with pytest", where I developed a very simple Python project together with the attendees following a TDD approach. It's a good way to introduce TDD, I think. I wrote each test together with the attendees, and then I left them the task of writing the Python code that passes the test. This way I could show TDD in action, introducing pytest features like the <code>pytest.raises</code> context manager or the use of <code>assert</code> while they become useful for the actual tests.</p>
<p>This is the approach that I follow in some of my posts on TDD here on the blog, for example <a href="https://www.thedigitalcatonline.com/blog/2015/05/13/python-oop-tdd-example-part1/">A simple example of Python OOP development (with TDD)</a> and <a href="https://www.thedigitalcatonline.com/blog/2017/05/09/a-game-of-tokens-write-an-interpreter-in-python-with-tdd-part-1/">A game of tokens: write an interpreter in Python with TDD - Part 1</a>.</p>
<p>Part of the workshop was dedicated to pytest command line options and in general to what pytest can do as a testing framework. Unfortunately there was no time to go through this part, so I promised some of the attendees to give them a written version of it. This post is the fulfilment of that promise.</p>
<p>Please remember to <code>import pytest</code> before using functions, decorators or attributes prefixed by <code>pytest.</code>, as I will not repeat it in each example.</p>
<h2 id="run-single-tests">Run single tests<a class="headerlink" href="#run-single-tests" title="Permanent link">¶</a></h2>
<p>If you want to run only a specific test you can provide its name on the pytest command line</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>-svv<span class="w"> </span>tests/test_calc.py::test_addition
</code></pre></div>
<p>which for example runs the <code>tests_addition</code> test inside the <code>tests/test_calc.py</code> file. You can also specify the file name only to run all the tests contained there</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>-svv<span class="w"> </span>tests/test_calc.py
</code></pre></div>
<h2 id="skipping-tests">Skipping tests<a class="headerlink" href="#skipping-tests" title="Permanent link">¶</a></h2>
<p>Sometimes it is useful to skip tests. The reason might be that some new code broke too many tests, and we want to face them one at a time, or that a specific feature had to be temporarily disabled. In all those cases the <code>pytest.mark.skip</code> decorator is your friend. Remember that a decorator is something that changes the way the decorated function works (for the skilled reader: it's a function wrapper). Assuming we are working on a <code>tests/test_calc.py</code> file the code might be</p>
<div class="highlight"><pre><span></span><code><span class="nd">@pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">skip</span>
<span class="k">def</span> <span class="nf">test_test_addition</span><span class="p">():</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
</code></pre></div>
<p>The result on the command line will be (after running <code>py.test -svv</code>)</p>
<div class="highlight"><pre><span></span><code>tests/test_calc.py::test_addition<span class="w"> </span>SKIPPED
</code></pre></div>
<h2 id="skipping-with-a-reason">Skipping with a reason<a class="headerlink" href="#skipping-with-a-reason" title="Permanent link">¶</a></h2>
<p>The previous solution is good for a temporary skip, but if the test has to remain deactivated for a long time it's better to annotate a specific reason for the exclusion. In my experience 1 day is enough to forget small details like this, so my advice is to always put a well-written reason on skipped tests. To add it you can use the <code>reason</code> attribute of the <code>skip</code> decorator</p>
<div class="highlight"><pre><span></span><code><span class="nd">@pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">skip</span><span class="p">(</span><span class="n">reason</span><span class="o">=</span><span class="s2">"Addition has been deactivated because of issue #123"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_test_addition</span><span class="p">():</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
</code></pre></div>
<p>Remember to add the <code>-rs</code> option to your command line to see the <code>r</code>eason behind <code>s</code>kipped tests. So after running <code>py.test -svv -rs</code> we will get something like</p>
<div class="highlight"><pre><span></span><code>tests/test_calc.py::test_addition<span class="w"> </span>SKIPPED
<span class="o">[</span>...<span class="o">]</span>
<span class="o">=============================</span><span class="w"> </span>short<span class="w"> </span><span class="nb">test</span><span class="w"> </span>summary<span class="w"> </span><span class="nv">info</span><span class="w"> </span><span class="o">=============================</span>
SKIP<span class="w"> </span><span class="o">[</span><span class="m">1</span><span class="o">]</span><span class="w"> </span>tests/test_calc.py:5:<span class="w"> </span>Addition<span class="w"> </span>has<span class="w"> </span>been<span class="w"> </span>deactivated<span class="w"> </span>because<span class="w"> </span>of<span class="w"> </span>issue<span class="w"> </span><span class="c1">#123</span>
<span class="o">======================</span><span class="w"> </span><span class="m">12</span><span class="w"> </span>passed,<span class="w"> </span><span class="m">1</span><span class="w"> </span>skipped<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.02<span class="w"> </span><span class="nv">seconds</span><span class="w"> </span><span class="o">=======================</span>
</code></pre></div>
<h2 id="skipping-tests-conditionally">Skipping tests conditionally<a class="headerlink" href="#skipping-tests-conditionally" title="Permanent link">¶</a></h2>
<p>Well, most of the time we will skip tests not for a stable reason, but according to some other condition that we can retrieve from the system, like the Python version, or maybe the region in which a server is running. The decorator that we need to use in that case is <code>skipif</code>, which accepts a condition (a boolean value) and a reason</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">os</span>
<span class="nd">@pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">skipif</span><span class="p">(</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s1">'AWS_REGION'</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'us-west-2'</span><span class="p">,</span>
<span class="n">reason</span><span class="o">=</span><span class="s2">"Addition has been deactivated in us-west-2 because of issue #234"</span>
<span class="p">)</span>
<span class="k">def</span> <span class="nf">test_addition</span><span class="p">():</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
</code></pre></div>
<p>With this code running <code>AWS_REGION=eu-west-1 py.test -svv -rs</code> will run the <code>test_addition</code> test, while running <code>AWS_REGION=us-west-2 py.test -svv -rs</code> will skip it. The environment variable <code>AWS_REGION</code> set in the previous command lines is an example that simulates the presence of the variable in the system.</p>
<h2 id="run-tests-by-name">Run tests by name<a class="headerlink" href="#run-tests-by-name" title="Permanent link">¶</a></h2>
<p>You can selectively run tests by name using <code>-k</code>. This option accepts Python expressions that try to match the name of the test with the provided values. So</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>-svv<span class="w"> </span>-k<span class="w"> </span><span class="s2">"test_addition"</span>
</code></pre></div>
<p>will run all those tests which name contains the substring <code>'addition'</code>, like <code>test_addiiton</code>, <code>test_addition_multiple_inputs</code>, and <code>test_complex_addition</code>. A more complex expression could be for example</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>-svv<span class="w"> </span>-k<span class="w"> </span><span class="s2">"test_addition and not complex"</span>
</code></pre></div>
<p>which will run both <code>test_addition</code> and <code>test_addition_multiple_inputs</code> but not <code>test_complex_addition</code>.</p>
<h2 id="tagging-tests">Tagging tests<a class="headerlink" href="#tagging-tests" title="Permanent link">¶</a></h2>
<p>Tests can be tagged or labelled using <code>pytest.mark</code>, and the tag can be used to run or skip sets of tests. Let's say that we identify a set of very slow tests that we don't want to run continuously.</p>
<div class="highlight"><pre><span></span><code><span class="nd">@pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">slow</span>
<span class="k">def</span> <span class="nf">test_addition</span><span class="p">():</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">test_subtraction</span><span class="p">():</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="nd">@pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">slow</span>
<span class="k">def</span> <span class="nf">test_multiplication</span><span class="p">():</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
</code></pre></div>
<p>In the above example <code>test_addition</code> and <code>test_multiplication</code> have been decorated with <code>pytest.mark.slow</code> which tells pytest to label them with the <code>slow</code> identifier. At this point we can run all the tests that are tagged with the <code>-m</code> option</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>-svv<span class="w"> </span>-m<span class="w"> </span>slow
</code></pre></div>
<p>Tests can be tagged multiple times</p>
<div class="highlight"><pre><span></span><code><span class="nd">@pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">complex</span>
<span class="nd">@pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">slow</span>
<span class="k">def</span> <span class="nf">test_addition</span><span class="p">():</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
</code></pre></div>
<p>In this case the test will be run both by <code>pytest -svv -m slow</code> and by <code>pytest -svv -m complex</code>.</p>
<p>The <code>-m</code> option supports complex expressions like</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>-svv<span class="w"> </span>-m<span class="w"> </span><span class="s1">'not slow'</span>
</code></pre></div>
<p>which runs all the tests that are not tagged with <code>slow</code>, or</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>-svv<span class="w"> </span>-m<span class="w"> </span><span class="s1">'mac or linux'</span>
</code></pre></div>
<p>which runs all the tests tagged with <code>mac</code> and all the tests tagged with <code>linux</code>. Pay attention that <code>-m</code> expressions refer to the tags of each single test, so <code>slow and complex</code> will run only those tests that are tagged both with <code>slow</code> and with <code>complex</code>, and not all the tests marked with the first and all the tests marked with the second.</p>
<h2 id="adding-a-command-line-option">Adding a command line option<a class="headerlink" href="#adding-a-command-line-option" title="Permanent link">¶</a></h2>
<p>You can add custom command line options to pytest with the <code>pytest_addoption</code> and <code>pytest_runtest_setup</code> hooks that allows you to manage the command line parser and the setup for each test.</p>
<p>Let's say, for example, that we want to add a <code>--runslow</code> option that runs all the tests marked with <code>slow</code>. First, create the file <code>tests/conftest.py</code>, which is a file that pytest imports before running your tests, and use the <code>pytest_addoption</code> hook</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">pytest_addoption</span><span class="p">(</span><span class="n">parser</span><span class="p">):</span>
<span class="n">parser</span><span class="o">.</span><span class="n">addoption</span><span class="p">(</span><span class="s2">"--runslow"</span><span class="p">,</span> <span class="n">action</span><span class="o">=</span><span class="s2">"store_true"</span><span class="p">,</span>
<span class="n">help</span><span class="o">=</span><span class="s2">"run slow tests"</span><span class="p">)</span>
</code></pre></div>
<p>The command line parser configuration will be stored into the <code>config</code> attribute of the setup of each test. Thus we can use the <code>pytest_runtest_setup</code> hook that runs before each test</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">pytest_runtest_setup</span><span class="p">(</span><span class="n">item</span><span class="p">):</span>
<span class="k">if</span> <span class="s1">'slow'</span> <span class="ow">in</span> <span class="n">item</span><span class="o">.</span><span class="n">keywords</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">item</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">getvalue</span><span class="p">(</span><span class="s2">"runslow"</span><span class="p">):</span>
<span class="n">pytest</span><span class="o">.</span><span class="n">skip</span><span class="p">(</span><span class="s2">"need --runslow option to run"</span><span class="p">)</span>
</code></pre></div>
<p>Here <code>item</code> is the single test, so <code>item.keywords</code> is the set of tags attached to the test, and <code>item.config</code> is the configuration after the parser run on the command line. This makes the previous code match all the tests that are decorated with <code>@pytest.mark.slow</code> and only when the <code>--runslow</code> option has been specified on the command line. If both those conditions are satisfied the <code>pytest.skip</code> function is run, which skips the current test adding the specified string as a reason.</p>
<h2 id="coverage">Coverage<a class="headerlink" href="#coverage" title="Permanent link">¶</a></h2>
<p>Coverage is a measure of the percentage of code lines are "hit" when running tests. Basically the idea is to discover if there are parts of the code that are not run during the tests, thus being untested.</p>
<p>If you follow a strict TDD methodology your coverage will be 100% always, because the only code you will write is the one that you need to pass the tests. But please, please, please keep this in mind: not everything is easily tested and in front of some complex parts of the code you always have to ask yourself "Is it worth?".</p>
<p>Is it worth spending 3 days to write a test for a feature? Well, if a failure in the new code means a huge financial loss for your company, yes. If you are writing a tool for yourself, and the code you are writing is not dangerous at all, maybe not. With all the shades of grey between these two black and white extreme cases.</p>
<p>So, don't become a slave of the coverage index. A coverage of more than 90% is heaven, and being over 80% is perfectly fine. I would say that, except for specific corner cases being under 80% means that you are not really following a TDD methodology. So, maybe go and review your work flow.</p>
<p>Anyway, pytest gives you a nice way to report the coverage using the <code>coverage</code> program. Just install <code>pytest-cov</code> with</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>pytest-cov
</code></pre></div>
<p>and run pytest with</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>-svv<span class="w"> </span>--cov<span class="o">=</span><name><span class="w"> </span>--cov-report<span class="o">=</span>term
</code></pre></div>
<p>where <code><name></code> is the name of the Python module you are testing (actually the path where the code you are testing is). This gives you a nice report with the percentage of covered code file by file.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>py.test<span class="w"> </span>-svv<span class="w"> </span>--cov<span class="o">=</span>mypymodule<span class="w"> </span>--cov-report<span class="o">=</span>term
-----------<span class="w"> </span>coverage:<span class="w"> </span>platform<span class="w"> </span>linux,<span class="w"> </span>python<span class="w"> </span><span class="m">3</span>.6.5-final-0<span class="w"> </span>-----------
Name<span class="w"> </span>Stmts<span class="w"> </span>Miss<span class="w"> </span>Cover
-------------------------------------------------
mypymodule/__init__.py<span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">100</span>%
mypymodule/calc.py<span class="w"> </span><span class="m">23</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">100</span>%
-------------------------------------------------
TOTAL<span class="w"> </span><span class="m">26</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">100</span>%
</code></pre></div>
<p>You may also use the <code>term-missing</code> report instad of just <code>term</code>, that lists the code blocks that are not covered</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>py.test<span class="w"> </span>-svv<span class="w"> </span>--cov<span class="o">=</span>mypymodule<span class="w"> </span>--cov-report<span class="o">=</span>term-missing
-----------<span class="w"> </span>coverage:<span class="w"> </span>platform<span class="w"> </span>linux,<span class="w"> </span>python<span class="w"> </span><span class="m">3</span>.6.5-final-0<span class="w"> </span>-----------
Name<span class="w"> </span>Stmts<span class="w"> </span>Miss<span class="w"> </span>Cover<span class="w"> </span>Missing
-----------------------------------------------------------
mypymodule/__init__.py<span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">100</span>%
mypymodule/calc.py<span class="w"> </span><span class="m">23</span><span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">91</span>%<span class="w"> </span><span class="m">6</span>,<span class="w"> </span><span class="m">11</span>
-----------------------------------------------------------
TOTAL<span class="w"> </span><span class="m">26</span><span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">92</span>%
</code></pre></div>
<p>Here I commented some of the tests to force the coverage percentage to drop. As you can see the report tells us that lines 6 and 11 of the <code>mypymodule/calc.py</code> file are not covered by any test.</p>
<h2 id="updates">Updates<a class="headerlink" href="#updates" title="Permanent link">¶</a></h2>
<p>2017-12-24: pytest.org (such an honour!) spotted a misspelled <code>pytest.mark.skip</code>. Thanks!</p>
<h2 id="feedback">Feedback<a class="headerlink" href="#feedback" title="Permanent link">¶</a></h2>
<p>Feel free to reach me on <a href="https://twitter.com/thedigicat">Twitter</a> if you have questions. The <a href="https://github.com/TheDigitalCatOnline/blog_source/issues">GitHub issues</a> page is the best place to submit corrections.</p>Clean architectures in Python: a step-by-step example2016-11-14T19:00:00+00:002021-09-24T12:00:00+00:00Leonardo Giordanitag:www.thedigitalcatonline.com,2016-11-14:/blog/2016/11/14/clean-architectures-in-python-a-step-by-step-example/<p>How to create software that can be easily changed and adapted, following the clean architecture principles</p><p>In 2015 I was introduced by my friend <a href="https://github.com/gekorob">Roberto Ciatti</a> to the concept of Clean Architecture, as it is called by Robert Martin. The well-known Uncle Bob talks a lot about this concept at conferences and wrote some very interesting posts about it. What he calls "Clean Architecture" is a way of structuring a software system, a set of consideration (more than strict rules) about the different layers and the role of the actors in it.</p><p>As he clearly states in a post aptly titled <a href="https://blog.8thlight.com/uncle-bob/2012/08/13/the-clean-architecture.html">The Clean Architecture</a>, the idea behind this design is not new. As a matter of fact, it is a set of concepts that have been pushed by many software engineers over the last 3 decades. One of the first implementations may be found in the Boundary-Control-Entity model proposed by Ivar Jacobson in his masterpiece <a href="https://www.ivarjacobson.com/publications/books/object-oriented-software-engineering-1992">Object-Oriented Software Engineering: A Use Case Driven Approach</a> published in 1992, but Martin lists other more recent versions of this architecture.</p><p>I will not repeat here what he had already explained better than I can do, so I will just point out some resources you may check to start exploring these concepts:</p><ul><li><a href="https://blog.8thlight.com/uncle-bob/2012/08/13/the-clean-architecture.html">The Clean Architecture</a> a post by Robert Martin that concisely describes the goals of the architecture. It also lists resources that describe similar architectures.</li><li><a href="https://blog.8thlight.com/uncle-bob/2014/05/12/TheOpenClosedPrinciple.html">The Open Closed Principle</a> a post by Robert Martin not strictly correlated with the Clean Architecture concept but important for the separation concept.</li><li>Hakka Labs: Robert "Uncle Bob" Martin - <a href="https://www.youtube.com/watch?v=HhNIttd87xs">Architecture: The Lost Years</a> a video of Robert Martin from Hakka Labs.</li><li><a href="http://www.taimila.com/blog/ddd-and-testing-strategy/">DDD & Testing Strategy</a> by Lauri Taimila</li><li><a href="https://www.amazon.co.uk/Clean-Architecture-Robert-C-Martin-x/dp/0134494164/ref=la_B000APG87E_1_3?s=books&ie=UTF8&qid=1479146201&sr=1-3">Clean Architecture</a> by Robert Martin, published by Prentice Hall.</li></ul><h2 id="a-day-in-the-life-of-a-clean-system-7792">A day in the life of a clean system<a class="headerlink" href="#a-day-in-the-life-of-a-clean-system-7792" title="Permanent link">¶</a></h2><p>I will introduce here a (very simple) system designed with a clean architecture. The purpose of this section is to familiarise with main concepts like <em>separation of concerns</em> and <em>inversion of control</em>, which are paramount in system design. While I describe how data flows in the system, I will purposefully omit details, so that we can focus on the global idea and not worry too much about the implementation.</p><h3 id="the-data-flow-a31a">The data flow</h3><p>In the rest of the book, we will design together part of a simple web application that provides a room renting system. So, let's consider that our "Rent-o-Matic" application (inspired by the Sludge-O-Matic™ from Day of the Tentacle) is running at <a href="https://www.rentomatic.com">https://www.rentomatic.com</a>, and that a user wants to see the available rooms. They open the browser and type the address, then clicking on menus and buttons they reach the page with the list of all the rooms that our company rents.</p><p>Let's assume that this URL is <code>/rooms?status=available</code>. When the user's browser accesses that URL, an HTTP request reaches our system, where there is a component that is waiting for HTTP connections. Let's call this component "web framework".</p><p>The purpose of the web framework is to understand the HTTP request and to retrieve the data that we need to provide a response. In this simple case there are two important parts of the request, namely the endpoint itself (<code>/rooms</code>), and a single query string parameter, <code>status=available</code>. Endpoints are like commands for our system, so when a user accesses one of them, they signal to the system that a specific service has been requested, which in this case is the list of all the rooms that are available for rent.</p><div class="imageblock"><img src="/images/cabook/figure01.svg"><div class="title">The web framework serving HTTP</div></div><p>The domain in which the web framework operates is that of the HTTP protocol, so when the web framework has decoded the request it should pass the relevant information to another component that will process it. This other component is called <em>use case</em>, and it is the crucial and most important component of the whole clean system as it implements the <em>business logic</em>.</p><div class="imageblock"><img src="/images/cabook/figure02.svg"><div class="title">The business logic</div></div><p>The business logic is an important concept in system design. You are creating a system because you have some knowledge that you think might be useful to the world, or at the very least marketable. This knowledge is, at the end of the day, a way to process data, a way to extract or present data that maybe others don't have. A search engine can find all the web pages that are related to the terms in a query, a social network shows you the posts of people you follow and sorts them according to a specific algorithm, a travel company finds the best options for your journey between two locations, and so on. All these are good examples of business logic.</p><div class="admonition tip"><i class="fa fa-lightbulb"></i><div class="content"><div class="title">Business logic</div><div><p>Business logic is the specific algorithm or process that you want to implement, the way you transform data to provide a service. It is the most important part of the system.</p></div></div></div><p>The use case implements a very specific part of the whole business logic. In this case we have a use case to search for rooms with a given value of the parameter <code>status</code>. This means that the use case has to extract all the rooms that are managed by our company and filter them to show only the ones that are available.</p><p>Why can't the web framework do it? Well, the main purpose of a good system architecture is to <em>separate concerns</em>, that is to keep different responsibilities and domains separated. The web framework is there to process the HTTP protocol, and is maintained by programmers that are concerned with that specific part of the system, and adding the business logic to it mixes two very different fields.</p><div class="admonition tip"><i class="fa fa-lightbulb"></i><div class="content"><div class="title">Separation of concerns</div><div><p>Different parts a system should manage different parts of the process. Whenever two separate parts of a system work on the same data or the same part of a process they are <em>coupled</em>. While coupling is unavoidable, the higher the coupling between two components the harder is to change one without affecting the other.</p></div></div></div><p>As we will see, separating layers allows us to maintain the system with less effort, making single parts of it more testable and easily replaceable.</p><p>In the example that we are discussing here, the use case needs to fetch all the rooms that are in an available state, extracting them from a source of data. This is the business logic, and in this case it is very straightforward, as it will probably consist of a simple filtering on the value of an attribute. This might however not be the case. An example of a more advanced business logic might be an ordering based on a recommendation system, which might require the use case to connect with more components than just the data source.</p><p>So, the information that the use case wants to process is stored somewhere. Let's call this component <em>storage system</em>. Many of you probably already pictured a database in your mind, maybe a relational one, but that is just one of the possible data sources. The abstraction represented by the storage system is: anything that the use case can access and that can provide data is a source. It might be a file, a database (either relational or not), a network endpoint, or a remote sensor.</p><div class="admonition tip"><i class="fa fa-lightbulb"></i><div class="content"><div class="title">Abstraction</div><div><p>When designing a system, it is paramount to think in terms of abstractions, or building blocks. A component has a role in the system, regardless of the specific implementation of that component. The higher the level of the abstraction, the less detailed are the components. Clearly, high-level abstractions don't consider practical problems, which is why the abstract design has to be then implemented using specific solutions or technologies.</p></div></div></div><p>For simplicity's sake, let's use a relational database like Postgres in this example, as it is likely to be familiar to the majority of readers, but keep in mind the more generic case.</p><div class="imageblock"><img src="/images/cabook/figure03.svg"><div class="title">The storage</div></div><p>How does the use case connect with the storage system? Clearly, if we hard code into the use case the calls to a specific system (e.g. using SQL) the two components will be <em>strongly coupled</em>, which is something we try to avoid in system design. Coupled components are not independent, they are tightly connected, and changes occurring in one of the two force changes in the second one (and vice versa). This also means that testing components is more difficult, as one component cannot live without the other, and when the second component is a complex system like a database this can severely slow down development.</p><p>For example, let's assume the use case called directly a specific Python library to access PostgreSQL such as <a href="https://www.psycopg.org/">psycopg</a>. This would couple the use case with that specific source, and a change of database would result in a change of its code. This is far from being ideal, as the use case contains the business logic, which has not changed moving from one database system to the other. Parts of the system that do not contain the business logic should be treated like implementation details.</p><div class="admonition tip"><i class="fa fa-lightbulb"></i><div class="content"><div class="title">Implementation detail</div><div><p>A specific solution or technology is called a <em>detail</em> when it is not central to the design as a whole. The word doesn't refer to the inherent complexity of the subject, which might be greater than that of more central parts.</p></div></div></div><p>A relational database is hundred of times richer and more complex than an HTTP endpoint, and this in turn is more complex than ordering a list of objects, but the core of the application is the use case, not the way we store data or the way we provide access to that. Usually, implementation details are mostly connected with performances or usability, while the core parts implement the pure business logic.</p><p>How can we avoid strong coupling? A simple solution is called <em>inversion of control</em>, and I will briefly sketch it here, and show a proper implementation in a later section of the book, when we will implement this very example.</p><p>Inversion of control happens in two phases. First, the called object (the database in this case) is wrapped with a standard interface. This is a set of functionalities shared by every implementation of the target, and each interface translates the functionalities to calls to the specific language<sup>[<a id="fr-692829ef-1" href="#">1</a>]</sup> of the wrapped implementation.</p><div class="admonition tip"><i class="fa fa-lightbulb"></i><div class="content"><div class="title">Inversion of control</div><div><p>A technique used to avoid strong coupling between components of a system, that involves wrapping them so that they expose a certain interface. A component expecting that interface can then connect to them without knowing the details of the specific implementation, and thus being strongly coupled to the interface instead of the specific implementation.</p></div></div></div><p>A real world example of this is that of power plugs: electric appliances are designed to be connected not with specific power plugs, but to any power plug that is build according to the specification (size, number of poles, etc). When you buy a TV in the UK, you expect it to come with a UK plug (BS 1363). If it doesn't, you need an <em>adapter</em> that allows you to plug electronic devices into sockets of a foreign nation. In this case, we need to connect the use case (TV) to a database (power system) that has not been designed to match a common interface.</p><p>In the example we are discussing, the use case needs to extract all rooms with a given status, so the database wrapper needs to provide a single entry point that we might call <code>list_rooms_with_status</code>.</p><div class="imageblock"><img src="/images/cabook/figure04.svg"><div class="title">The storage interface</div></div><p>In the second phase of inversion of control the caller (the use case) is modified to avoid hard coding the call to the specific implementation, as this would again couple the two. The use case accepts an incoming object as a parameter of its constructor, and receives a concrete instance of the adapter at creation time. The specific technique used to implement this depends greatly on the programming language we use. Python doesn't have an explicit syntax for interfaces, so we will just assume the object we pass implements the required methods.</p><div class="imageblock"><img src="/images/cabook/figure05.svg"><div class="title">Inversion of control on the storage interface</div></div><p>Now the use case is connected with the adapter and knows the interface, and it can call the entry point <code>list_rooms_with_status</code> passing the status <code>available</code>. The adapter knows the details of the storage system, so it converts the method call and the parameter in a specific call (or set of calls) that extract the requested data, and then converts them in the format expected by the use case. For example, it might return a Python list of dictionaries that represent rooms.</p><div class="imageblock"><img src="/images/cabook/figure06.svg"><div class="title">The business logic extracts data from the storage</div></div><p>At this point, the use case has to apply the rest of the business logic, if needed, and return the result to the web framework.</p><div class="imageblock"><img src="/images/cabook/figure07.svg"><div class="title">The business logic returns processed data to the web framework</div></div><p>The web framework converts the data received from the use case into an HTTP response. In this case, as we are considering an endpoint that is supposed to be reached explicitly by the user of the website, the web framework will return an HTML page in the body of the response, but if this was an internal endpoint, for example called by some asynchronous JavaScript code in the front-end, the body of the response would probably just be a JSON structure.</p><div class="imageblock"><img src="/images/cabook/figure08.svg"><div class="title">The web framework returns the data in an HTTP response</div></div><h3 id="advantages-of-a-layered-architecture-a271">Advantages of a layered architecture</h3><p>As you can see, the stages of this process are clearly separated, and there is a great deal of data transformation between them. Using common data formats is one of the way we achieve independence, or loose coupling, between components of a computer system.</p><p>To better understand what loose coupling means for a programmer, let's consider the last picture. In the previous paragraphs I gave an example of a system that uses a web framework for the user interface and a relational database for the data source, but what would change if the front-end part was a command-line interface? And what would change if, instead of a relational database, there was another type of data source, for example a set of text files?</p><div class="imageblock"><img src="/images/cabook/figure09.svg"><div class="title">The web framework replaced by a CLI</div></div><div class="imageblock"><img src="/images/cabook/figure10.svg"><div class="title">A database replaced by a more trivial file-based storage</div></div><p>As you can see, both changes would require the replacement of some components. After all, we need different code to manage a command line instead of a web page. But the external shape of the system doesn't change, neither does the way data flows. We created a system in which the user interface (web framework, command-line interface) and the data source (relational database, text files) are details of the implementation, and not core parts of it.</p><p>The main immediate advantage of a layered architecture, however, is testability. When you clearly separate components you clearly establish the data each of them has to receive and produce, so you can ideally disconnect a single component and test it in isolation. Let's take the Web framework component that we added and consider it for a moment forgetting the rest of the architecture. We can ideally connect a tester to its inputs and outputs as you can see in the figure</p><div class="imageblock"><img src="/images/cabook/figure11.svg"><div class="title">Testing the web layer in isolation</div></div><div class="imageblock"><img src="/images/cabook/figure12.svg"><div class="title">Detailed setup of the web layer testing</div></div><p>We know that the Web framework receives an HTTP request <span class="callout">1</span> with a specific target and a specific query string, and that it has to call <span class="callout">2</span> a method on the use case passing specific parameters. When the use case returns data <span class="callout">3</span>, the Web framework has to convert that into an HTTP response <span class="callout">4</span>. Since this is a test we can have a fake use case, that is an object that just mimics what the use case does without really implementing the business logic. We will then test that the Web framework calls the method <span class="callout">2</span> with the correct parameters, and that the HTTP response <span class="callout">4</span> contains the correct data in the proper format, and all this will happen without involving any other part of the system.</p>
<div class="advertisement">
<a href="https://www.thedigitalcat.academy/freebie-first-class-objects">
<img src="/images/first-class-objects/cover.jpg" />
</a>
<div class="body">
<h2 id="first-class-objects-in-python-fffa">First-class objects in Python<a class="headerlink" href="#first-class-objects-in-python-fffa" title="Permanent link">¶</a></h2>
<p>Higher-order functions, wrappers, and factories</p>
<p>Learn all you need to know to understand first-class citizenship in Python, the gateway to grasp how decorators work and how functional programming can supercharge your code.</p>
<div class="actions">
<a class="action" href="https://www.thedigitalcat.academy/freebie-first-class-objects">Get your FREE copy</a>
</div>
</div>
</div>
<h2 id="clean-architectures-in-python-the-book-c4fa">Clean Architectures in Python: the book<a class="headerlink" href="#clean-architectures-in-python-the-book-c4fa" title="Permanent link">¶</a></h2><p>I hope you found this introduction useful. What you read so far was the first chapter of the book "Clean Architectures in Python" that you can read online at <a href="https://www.thedigitalcatbooks.com/pycabook-introduction/">The Digital Cat Books</a>. The book is available as PDF and ebook <a href="https://leanpub.com/clean-architectures-in-python">on Leanpub</a>.</p><div class="imageblock"><img src="/images/cabook/cover.jpg"></div><p>Chapter 2 of the book briefly discusses the <strong>components</strong> and the ideas behind this software architecture. Chapter 3 runs through <strong>a concrete example</strong> of clean architecture and chapter 4 expands the example adding a <strong>web application</strong> on top of it. Chapter 5 discusses <strong>error management</strong> and improvements to the Python code developed in the previous chapters. Chapters 6 and 7 show how to plug <strong>different database systems</strong> to the web service created previously, and chapter 8 wraps up the example showing how to run the application with a <strong>production-ready configuration</strong>.</p><h2 id="feedback-d845">Feedback<a class="headerlink" href="#feedback-d845" title="Permanent link">¶</a></h2><p>Feel free to reach me on <a href="https://twitter.com/thedigicat">Twitter</a> if you have questions. The <a href="https://github.com/TheDigitalCatOnline/blog_source/issues">GitHub issues</a> page is the best place to submit corrections.</p>Python Mocks: a gentle introduction - Part 22016-09-27T09:00:00+00:002016-09-27T09:00:00+00:00Leonardo Giordanitag:www.thedigitalcatonline.com,2016-09-27:/blog/2016/09/27/python-mocks-a-gentle-introduction-part-2/<p>Second part of the series on Python mocks: how to patch code</p><p>In the <a href="https://www.thedigitalcatonline.com/blog/2016/03/06/python-mocks-a-gentle-introduction-part-1/">first post</a> I introduced you to Python mocks, objects that can imitate other objects and work as placeholders, replacing external systems during unit testing. I described the basic behaviour of mock objects, the <code>return_value</code> and <code>side_effect</code> attributes, and the <code>assert_called_with()</code> method.</p>
<p>In this post I will briefly review the remaining <code>assert_*</code> methods and some interesting attributes that allow to check the calls received by the mock object. Then I will introduce and exemplify patching, which is a very important topic in testing.</p>
<h2 id="other-assertions-and-attributes">Other assertions and attributes<a class="headerlink" href="#other-assertions-and-attributes" title="Permanent link">¶</a></h2>
<p>The <a href="https://docs.python.org/dev/library/unittest.mock.html">official documentation</a> of the mock library lists many other assertion, namely <code>assert_called_once_with()</code>, <code>assert_any_call()</code>, <code>assert_has_calls()</code>, <code>assert_not_called()</code>. If you grasped how <code>assert_called_with()</code> works, you will have no troubles in understanding how those other behave. Be sure to check the documentation to get a full description of what mock object can assert about their history after being used by your code.</p>
<p>Together with those methods, mock objects also provide some useful attributes, two of which have been already reviewed in the first post. The remaining attributes are as expected mostly related to calls, and are <code>called</code>, <code>call_count</code>, <code>call_args</code>, <code>call_args_list</code>, <code>method_calls</code>, <code>mock_calls</code>. While these also are very well described in the official documentation, I want to point out the two <code>method_calls</code> and <code>mock_calls</code> attributes, that store the detailed list of methods which are called on the mock, and the <code>call_args_list</code> attribute that lists the parameters of every call.</p>
<p>Do not forget that methods called on a mock object are mocks themselves, so you may first access the main mock object to get information about the called methods, and then access those methods to get the arguments they received.</p>
<h2 id="patching">Patching<a class="headerlink" href="#patching" title="Permanent link">¶</a></h2>
<p>Mocks are very simple to introduce in your tests whenever your objects accept classes or instances from outside. In that case, as described, you just have to instantiate the <code>Mock</code> class and pass the resulting object to your system. However, when the external classes instantiated by your library are hardcoded this simple trick does not work. In this case you have no chance to pass a mock object instead of the real one.</p>
<p>This is exactly the case addressed by patching. Patching, in a testing framework, means to replace a globally reachable object with a mock, thus achieving the target of having the code run unmodified, while part of it has been hot swapped, that is, replaced at run time.</p>
<h2 id="a-warm-up-example">A warm-up example<a class="headerlink" href="#a-warm-up-example" title="Permanent link">¶</a></h2>
<p>Let us start with a very simple example. Patching can be complex to grasp at the beginning so it is better to learn it with trivial code. If you do not have it yet, create the testing environment <code>mockplayground</code> with the instruction given in the previous post.</p>
<p>I want to develop a simple class that returns information about a given file. The class shall be instantiated with the file name, which can be a relative path.</p>
<p>For the sake of brevity I will not show you every step of the TDD development of the class. Remember that TDD requires you to write a test and then implement the code, but sometimes this could be too fine grained, so do not use the TDD rules without thinking.</p>
<p>The tests for the initialization of the class are</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">fileinfo</span> <span class="kn">import</span> <span class="n">FileInfo</span>
<span class="k">def</span> <span class="nf">test_init</span><span class="p">():</span>
<span class="n">filename</span> <span class="o">=</span> <span class="s1">'somefile.ext'</span>
<span class="n">fi</span> <span class="o">=</span> <span class="n">FileInfo</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">fi</span><span class="o">.</span><span class="n">filename</span> <span class="o">==</span> <span class="n">filename</span>
<span class="k">def</span> <span class="nf">test_init</span><span class="p">():</span>
<span class="n">filename</span> <span class="o">=</span> <span class="s1">'somefile.ext'</span>
<span class="n">relative_path</span> <span class="o">=</span> <span class="s1">'../</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="n">fi</span> <span class="o">=</span> <span class="n">FileInfo</span><span class="p">(</span><span class="n">relative_path</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">fi</span><span class="o">.</span><span class="n">filename</span> <span class="o">==</span> <span class="n">filename</span>
</code></pre></div>
<p>You can put them into the <code>tests/test_fileinfo.py</code> file. The code that makes the tests pass could be something like</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">os</span>
<span class="k">class</span> <span class="nc">FileInfo</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">original_path</span> <span class="o">=</span> <span class="n">path</span>
<span class="bp">self</span><span class="o">.</span><span class="n">filename</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">basename</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
</code></pre></div>
<p>Up to now I didn't introduce any new feature. Now I want the <code>get_info()</code> function to return a tuple with the file name, the original path the class was instantiated with, and the absolute path of the file.</p>
<p>You immediately realise that you have an issue in writing the test. There is no way to easily test something as "the absolute path", since the outcome of the function called in the test is supposed to vary with the path of the test itself. Let us write part of the test</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">test_get_info</span><span class="p">():</span>
<span class="n">filename</span> <span class="o">=</span> <span class="s1">'somefile.ext'</span>
<span class="n">original_path</span> <span class="o">=</span> <span class="s1">'../</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="n">fi</span> <span class="o">=</span> <span class="n">FileInfo</span><span class="p">(</span><span class="n">original_path</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">fi</span><span class="o">.</span><span class="n">get_info</span><span class="p">()</span> <span class="o">==</span> <span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">original_path</span><span class="p">,</span> <span class="s1">'???'</span><span class="p">)</span>
</code></pre></div>
<p>where the '???' string highlights that I cannot put something sensible to test the absolute path of the file.</p>
<p>Patching is the way to solve this problem. You know that the function will use some code to get the absolute path of the file. So in the scope of the test only you can replace that code with different code and perform the test. Since the replacement code has a known outcome writing the test is now possible.</p>
<p>Patching, thus, means to inform Python that in some scope you want a globally accessible module/object replaced by a mock. Let's see how we can use it in our example</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">unittest.mock</span> <span class="kn">import</span> <span class="n">patch</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">test_get_info</span><span class="p">():</span>
<span class="n">filename</span> <span class="o">=</span> <span class="s1">'somefile.ext'</span>
<span class="n">original_path</span> <span class="o">=</span> <span class="s1">'../</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="k">with</span> <span class="n">patch</span><span class="p">(</span><span class="s1">'os.path.abspath'</span><span class="p">)</span> <span class="k">as</span> <span class="n">abspath_mock</span><span class="p">:</span>
<span class="n">test_abspath</span> <span class="o">=</span> <span class="s1">'some/abs/path'</span>
<span class="n">abspath_mock</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="n">test_abspath</span>
<span class="n">fi</span> <span class="o">=</span> <span class="n">FileInfo</span><span class="p">(</span><span class="n">original_path</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">fi</span><span class="o">.</span><span class="n">get_info</span><span class="p">()</span> <span class="o">==</span> <span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">original_path</span><span class="p">,</span> <span class="n">test_abspath</span><span class="p">)</span>
</code></pre></div>
<p>Remember that if you are using Python 2 you installed the <code>mock</code> module with <code>pip</code>, so your import statement becomes <code>from mock import patch</code>.</p>
<p>You clearly see the context in which the patching happens, as it is enclosed in a <code>with</code> statement. Inside this statement the module <code>os.path.abspath</code> will be replaced by a mock created by the function <code>patch</code> and called <code>abspath_mock</code>. We can now give the function a <code>return_value</code> as we did with standard mocks in the first post and run the test.</p>
<p>The code that make the test pass is</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">FileInfo</span><span class="p">:</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">get_info</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">filename</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">original_path</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">abspath</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">filename</span><span class="p">)</span>
</code></pre></div>
<p>Obviously to write the test you have to know that you are going to use the <code>os.path.abspath</code> function, so patching is somehow a "less pure" practice in TDD. In pure OOP/TDD you are only concerned with the external behaviour of the object, and not with its internal structure. This example, however, shows that you have to cope with some real world issues, and patching is a clean way to do it.</p>
<h2 id="the-patching-decorator">The patching decorator<a class="headerlink" href="#the-patching-decorator" title="Permanent link">¶</a></h2>
<p>The <code>patch</code> function we imported from the <code>unittest.mock</code> module is very powerful, and can be used as a function decorator as well. When used in this fashion you need to change the decorated function to accept a mock as last argument.</p>
<div class="highlight"><pre><span></span><code><span class="nd">@patch</span><span class="p">(</span><span class="s1">'os.path.abspath'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_get_info</span><span class="p">(</span><span class="n">abspath_mock</span><span class="p">):</span>
<span class="n">filename</span> <span class="o">=</span> <span class="s1">'somefile.ext'</span>
<span class="n">original_path</span> <span class="o">=</span> <span class="s1">'../</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="n">test_abspath</span> <span class="o">=</span> <span class="s1">'some/abs/path'</span>
<span class="n">abspath_mock</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="n">test_abspath</span>
<span class="n">fi</span> <span class="o">=</span> <span class="n">FileInfo</span><span class="p">(</span><span class="n">original_path</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">fi</span><span class="o">.</span><span class="n">get_info</span><span class="p">()</span> <span class="o">==</span> <span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">original_path</span><span class="p">,</span> <span class="n">test_abspath</span><span class="p">)</span>
</code></pre></div>
<p>As you can see the <code>patch</code> decorator works like a big <code>with</code> statement for the whole function. Obviously in this way you replace the target function <code>os.path.abspath</code> in the scope of the whole function. It is then up to you to decide if you need to use <code>patch</code> as a decorator or in a <code>with</code> block.</p>
<h2 id="multiple-patches">Multiple patches<a class="headerlink" href="#multiple-patches" title="Permanent link">¶</a></h2>
<p>We can also patch more that one object. Say for example that we want to change the above test to check that the outcome of the <code>FileInfo.get_info()</code> method also contains the size of the file. To get the size of a file in Python we may use the <code>os.path.getsize()</code> function, which returns the size of the file in bytes.</p>
<p>So now we have to patch <code>os.path.getsize</code> as well, and this can be done with another <code>patch</code> decorator.</p>
<div class="highlight"><pre><span></span><code><span class="nd">@patch</span><span class="p">(</span><span class="s1">'os.path.getsize'</span><span class="p">)</span>
<span class="nd">@patch</span><span class="p">(</span><span class="s1">'os.path.abspath'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_get_info</span><span class="p">(</span><span class="n">abspath_mock</span><span class="p">,</span> <span class="n">getsize_mock</span><span class="p">):</span>
<span class="n">filename</span> <span class="o">=</span> <span class="s1">'somefile.ext'</span>
<span class="n">original_path</span> <span class="o">=</span> <span class="s1">'../</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="n">test_abspath</span> <span class="o">=</span> <span class="s1">'some/abs/path'</span>
<span class="n">abspath_mock</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="n">test_abspath</span>
<span class="n">test_size</span> <span class="o">=</span> <span class="mi">1234</span>
<span class="n">getsize_mock</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="n">test_size</span>
<span class="n">fi</span> <span class="o">=</span> <span class="n">FileInfo</span><span class="p">(</span><span class="n">original_path</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">fi</span><span class="o">.</span><span class="n">get_info</span><span class="p">()</span> <span class="o">==</span> <span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">original_path</span><span class="p">,</span> <span class="n">test_abspath</span><span class="p">,</span> <span class="n">test_size</span><span class="p">)</span>
</code></pre></div>
<p>Please notice that the decorator which is nearest to the function is applied first. Always remember that the decorator syntax with <code>@</code> is a shortcut to replace the function with the output of the decorator, so two decorators result in</p>
<div class="highlight"><pre><span></span><code><span class="nd">@decorator1</span>
<span class="nd">@decorator2</span>
<span class="k">def</span> <span class="nf">myfunction</span><span class="p">():</span>
<span class="k">pass</span>
</code></pre></div>
<p>which is a shorcut for</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">myfunction</span><span class="p">():</span>
<span class="k">pass</span>
<span class="n">myfunction</span> <span class="o">=</span> <span class="n">decorator1</span><span class="p">(</span><span class="n">decorator2</span><span class="p">(</span><span class="n">myfunction</span><span class="p">))</span>
</code></pre></div>
<p>This explains why, in the test code, the function receives first <code>abspath_mock</code> and then <code>getsize_mock</code>. The first decorator applied to the function is the patch of <code>os.path.abspath</code>, which appends the mock that we call <code>abspath_mock</code>. Then the patch of <code>os.path.getsize</code> is applied and this appends its own mock.</p>
<p>The code that makes the test pass is</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">FileInfo</span><span class="p">:</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">get_info</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">filename</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">original_path</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">abspath</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">filename</span><span class="p">),</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">getsize</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">filename</span><span class="p">)</span>
</code></pre></div>
<p>We can write the above test using two <code>with</code> statements as well</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">test_get_info</span><span class="p">():</span>
<span class="n">filename</span> <span class="o">=</span> <span class="s1">'somefile.ext'</span>
<span class="n">original_path</span> <span class="o">=</span> <span class="s1">'../</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="k">with</span> <span class="n">patch</span><span class="p">(</span><span class="s1">'os.path.abspath'</span><span class="p">)</span> <span class="k">as</span> <span class="n">abspath_mock</span><span class="p">:</span>
<span class="n">test_abspath</span> <span class="o">=</span> <span class="s1">'some/abs/path'</span>
<span class="n">abspath_mock</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="n">test_abspath</span>
<span class="k">with</span> <span class="n">patch</span><span class="p">(</span><span class="s1">'os.path.getsize'</span><span class="p">)</span> <span class="k">as</span> <span class="n">getsize_mock</span><span class="p">:</span>
<span class="n">test_size</span> <span class="o">=</span> <span class="mi">1234</span>
<span class="n">getsize_mock</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="n">test_size</span>
<span class="n">fi</span> <span class="o">=</span> <span class="n">FileInfo</span><span class="p">(</span><span class="n">original_path</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">fi</span><span class="o">.</span><span class="n">get_info</span><span class="p">()</span> <span class="o">==</span> <span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">original_path</span><span class="p">,</span> <span class="n">test_abspath</span><span class="p">,</span> <span class="n">test_size</span><span class="p">)</span>
</code></pre></div>
<p>Using more than one <code>with</code> statement, however, makes the code difficult to read, in my opinion, so in general I prefer to avoid complex <code>with</code> trees if I do not need a limited scope of the patching.</p>
<h2 id="patching-immutable-objects">Patching immutable objects<a class="headerlink" href="#patching-immutable-objects" title="Permanent link">¶</a></h2>
<p>The most widespread version of Python is CPython, which is written, as the name suggests, in C. Part of the standard library is also written in C, while the rest is written in Python itself.</p>
<p>The objects (classes, modules, functions, etc) that are implemented in C are shared between interpreters, which is something that you can do embedding the Python interpreter in a C program, for example. This requires those objects to be immutable, so that you cannot alter them at runtime from a single interpreter.</p>
<p>For an example of this immutability just check the following code</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">a</span> <span class="o">=</span> <span class="mi">1</span>
<span class="gp">>>> </span><span class="n">a</span><span class="o">.</span><span class="n">conjugate</span> <span class="o">=</span> <span class="mi">5</span>
<span class="gt">Traceback (most recent call last):</span>
File <span class="nb">"<stdin>"</span>, line <span class="m">1</span>, in <span class="n"><module></span>
<span class="gr">AttributeError</span>: <span class="n">'int' object attribute 'conjugate' is read-only</span>
</code></pre></div>
<p>Here I'm trying to replace a method with an integer, which is pointless, but nevertheless shows the issue we are facing.</p>
<p>What has this immutability to do with patching? What <code>patch</code> does is actually to temporarily replace an attibute of an object (method of a class, class of a module, etc), so if that object is immutable the patching action fails.</p>
<p>A typical example of this problem is the <code>datetime</code> module, which is also one of the best candidates for patching, since the output of time functions is by definition time-varying.</p>
<p>Let me show the problem with a simple class that logs operations. The class is the following (you can put it into a file called <code>logger.py</code>)</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">datetime</span>
<span class="k">class</span> <span class="nc">Logger</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">messages</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">def</span> <span class="nf">log</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">messages</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">(),</span> <span class="n">message</span><span class="p">))</span>
</code></pre></div>
<p>This is pretty simple, but testing this code is problematic, because the <code>log()</code> method produces results that depend on the actual execution time.</p>
<p>If we try to write a test patching <code>datetime.datetime.now</code> we have a bitter surprise. This is the test code, that you can put in <code>tests/test_logger.py</code></p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">unittest.mock</span> <span class="kn">import</span> <span class="n">patch</span>
<span class="kn">from</span> <span class="nn">logger</span> <span class="kn">import</span> <span class="n">Logger</span>
<span class="k">def</span> <span class="nf">test_init</span><span class="p">():</span>
<span class="n">l</span> <span class="o">=</span> <span class="n">Logger</span><span class="p">()</span>
<span class="k">assert</span> <span class="n">l</span><span class="o">.</span><span class="n">messages</span> <span class="o">==</span> <span class="p">[]</span>
<span class="nd">@patch</span><span class="p">(</span><span class="s1">'datetime.datetime.now'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_log</span><span class="p">(</span><span class="n">mock_now</span><span class="p">):</span>
<span class="n">test_now</span> <span class="o">=</span> <span class="mi">123</span>
<span class="n">test_message</span> <span class="o">=</span> <span class="s2">"A test message"</span>
<span class="n">mock_now</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="n">test_now</span>
<span class="n">l</span> <span class="o">=</span> <span class="n">Logger</span><span class="p">()</span>
<span class="n">l</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="n">test_message</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">l</span><span class="o">.</span><span class="n">messages</span> <span class="o">==</span> <span class="p">[(</span><span class="n">test_now</span><span class="p">,</span> <span class="n">test_message</span><span class="p">)]</span>
</code></pre></div>
<p>and the execution of pytest returns a <code>TypeError: can't set attributes of built-in/extension type 'datetime.datetime'</code>, which is exactly a problem of immutability.</p>
<p>There are several ways to address this problem, but all of them leverage the fact that, when you import of subclass an immutable object what you get is a "copy" of that is now mutable.</p>
<p>The easiest example in this case is the module <code>datetime</code> itself. In the <code>test_log</code> function we try to patch directly the <code>datetime.datetime.now</code> object, affecting the builtin module <code>datetime</code>. The file <code>logger.py</code>, however, does import <code>datetime</code>, so that the latter becomes a local symbol in the <code>logger</code> module. This is exactly the key for our patching. Let us change the code to</p>
<div class="highlight"><pre><span></span><code><span class="nd">@patch</span><span class="p">(</span><span class="s1">'logger.datetime.datetime'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_log</span><span class="p">(</span><span class="n">mock_datetime</span><span class="p">):</span>
<span class="n">test_now</span> <span class="o">=</span> <span class="mi">123</span>
<span class="n">test_message</span> <span class="o">=</span> <span class="s2">"A test message"</span>
<span class="n">mock_datetime</span><span class="o">.</span><span class="n">now</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="n">test_now</span>
<span class="n">l</span> <span class="o">=</span> <span class="n">Logger</span><span class="p">()</span>
<span class="n">l</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="n">test_message</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">l</span><span class="o">.</span><span class="n">messages</span> <span class="o">==</span> <span class="p">[(</span><span class="n">test_now</span><span class="p">,</span> <span class="n">test_message</span><span class="p">)]</span>
</code></pre></div>
<p>As you see running the test now the patching works. What we did was to patch <code>logger.datetime.datetime</code> instead of <code>datetime.datetime.now</code>. Two things changed, thus, in our test. First, we are patching the module imported in the <code>logger.py</code> file and not the module provided globally by the Python interpreter. Second, we have to patch the whole module because this is what is imported by the <code>logger.py</code> file. If you try to patch <code>logger.datetime.datetime.now</code> you will find that it is still immutable.</p>
<p>Another possible solution to this problem is to create a function that invokes the immutable object and returns its value. This last function can be easily patched, because it just uses the builtin objects and thus is not immutable. This solution, however, requires to change the source code to allow testing, which is far from being desirable. Obviously it is better to introduce a small change in the code and have it tested than to leave it untested, but whenever is possible I avoid solutions that introduce code which wouldn't be required without tests.</p>
<h2 id="final-words">Final words<a class="headerlink" href="#final-words" title="Permanent link">¶</a></h2>
<p>In this second part of this small series on Python testing we reviewed the patching mechanism and run through some of its subtleties. Patching is a really effective technique, and patch-based tests can be found in many different packages. Take your time to become confident with mocks and patching, since they will be one of your main tools while working with Python and any other object-oriented language.</p>
<p>As always, I strongly recommend finding some time to read the <a href="https://docs.python.org/dev/library/unittest.mock.html">official documentation</a> of the mock library.</p>
<h2 id="feedback">Feedback<a class="headerlink" href="#feedback" title="Permanent link">¶</a></h2>
<p>The <a href="https://github.com/TheDigitalCatOnline/blog_source/issues">GitHub issues</a> page is the best place to submit corrections</p>Punch - Update your version while having a drink2016-05-31T16:00:00+01:002016-05-31T16:00:00+01:00Leonardo Giordanitag:www.thedigitalcatonline.com,2016-05-31:/blog/2016/05/31/punch-update-your-version-while-having-a-drink/<p>So you completed your wonderful new project, all your test are successful (you <a href="/categories/tdd/">test</a> code, don't you?) and you just want to ship the new version and call it a day. Well, you just have to go and change the version number in your install script and save. Oh, right, you also have to open a feature branch, so that you may record the version update in your Git history. Well, easily done. Damn! You forgot to change the version number in the README.md file...</p>
<p>Managing the version number of a project is not easy. Not only you need to think about the versioning scheme and what part of the version to increase (see <a href="https://www.thedigitalcatonline.com/blog/2013/03/20/versioning-an-underrated-discipline/">this post</a> for some tips on this matter), but you also need to remember in which files you put the actual version number, and, depending on your workflow, to correctly manage the version control system commits.</p>
<p><strong>Punch</strong> is a small tool that aims to simplify the latter parts, that is the management of the version number update process. Punch is a young project but the underlying structure should be flexible enough to allow the implementation of advanced usage cases that may arise in the future.</p>
<p><img alt="Punch logo" src="/images/punch/icon_400x400.png"></p>
<h4 id="features">Features<a class="headerlink" href="#features" title="Permanent link">¶</a></h4>
<ul>
<li>Written in Python, but manages every kind of text file</li>
<li>May implement different versioning schemas</li>
<li>Currently supports integer version parts and multiple value integer parts</li>
<li>Version parts may be explicitly set to given values</li>
<li>The same version number can be represented in different ways using Jinja2 templates</li>
<li>Each managed file may override the global behaviour with custom settings</li>
<li>Can automatically commit using Git or git-flow</li>
<li>May optionally create annotated commits</li>
<li>Can simulate the version upgrade and show what is going to happen</li>
</ul>
<h4 id="installation">Installation<a class="headerlink" href="#installation" title="Permanent link">¶</a></h4>
<p>You may install Punch in your virtual environment (or in your system) directly from PyPI</p>
<div class="highlight"><pre><span></span><code>pip<span class="w"> </span>install<span class="w"> </span>punch.py
</code></pre></div>
<p>and you will have the <code>punch</code> executable available.</p>
<h4 id="useful-links">Useful links<a class="headerlink" href="#useful-links" title="Permanent link">¶</a></h4>
<ul>
<li>Punch official documentation: <a href="https://punch.readthedocs.io/en/latest/">https://punch.readthedocs.io/en/latest/</a></li>
<li>Official repository <a href="https://github.com/lgiordani/punch">https://github.com/lgiordani/punch</a></li>
<li>Issues and pull requests <a href="https://github.com/lgiordani/punch/issues">https://github.com/lgiordani/punch/issues</a></li>
</ul>
<p>Feel free to submit your issues or pull requests, contributions are welcome.</p>Abstract Base Classes in Python2016-04-03T11:00:00+01:002016-04-03T11:00:00+01:00Leonardo Giordanitag:www.thedigitalcatonline.com,2016-04-03:/blog/2016/04/03/abstract-base-classes-in-python/<p>What are Abstract Base Classes in Python and why are they useful?</p><p>With the introduction of Abstract Base Classes, Python once again shows its nature of a very innovative and flexible language. It is interesting to see how such a remarkable feature has been introduced into the language by a pure Python module. This demonstrates that Python is built in a way that is very open to changes, thanks to its foundations in pure polymorphism based on delegation.</p>
<p>Many Python programmers overlooked Abstract Base Classes and the classes in the <code>collections</code> module, which are one of the simplest and useful applications of the concept. Sure enough, this is not a feature that you will use every day or that will change the way you are programming in Python. But neither is it something you shall discard before understanding what it brings into the language, and what sort of problems it can solve for you.</p>
<h2 id="eafp">EAFP<a class="headerlink" href="#eafp" title="Permanent link">¶</a></h2>
<p>Python is a dynamically-typed object-oriented language strongly based on delegation, so its approach to problems is intrinsically polymorphic. This means that Python deals mostly with the behaviour of objects and not with their structure. The well-known EAFP protocol (it's Easier to Ask Forgiveness than Permission) comes from this approach. This code</p>
<div class="highlight"><pre><span></span><code><span class="k">try</span><span class="p">:</span>
<span class="n">someobj</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="k">except</span> <span class="ne">TypeError</span><span class="p">:</span>
<span class="c1"># object is not subscriptable</span>
<span class="o">...</span>
</code></pre></div>
<p>does not check if the object is a <code>list</code> or a <code>dictionary</code> (both would allow the <code>[1]</code> notation), but if the object can be accessed by key (or index). When you accept a parameter in a function Python does not specify the type (leaving aside type hints) because you are not interested in accepting a given type or one of its derived types. You are interested in accepting something that provides the methods you will use.</p>
<p>The behaviour, in an object-oriented environment, is the run-time interface of the object. This is different from the static interface, which is the collection of the methods provided by the object. The run-time interface is the actual interface the object shows when it is used, and this encompasses the methods provided by its class, but also methods provided by parent classes, the metaclass and other entry points provided by <code>__getattr__</code>.</p>
<h2 id="complex-checks">Complex checks<a class="headerlink" href="#complex-checks" title="Permanent link">¶</a></h2>
<p>Sometimes, however, you need to perform complex checks, such as "it behaves like a list". How can you test this condition? You could test if the incoming object has some standard methods, but this is not only incomplete but also wrong. For example, I could write the following test</p>
<div class="highlight"><pre><span></span><code><span class="k">try</span><span class="p">:</span>
<span class="n">obj</span><span class="o">.</span><span class="n">append</span>
<span class="n">obj</span><span class="o">.</span><span class="n">count</span>
<span class="n">obj</span><span class="o">.</span><span class="n">extend</span>
<span class="n">obj</span><span class="o">.</span><span class="n">index</span>
<span class="n">obj</span><span class="o">.</span><span class="n">insert</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="k">except</span> <span class="ne">AttributeError</span><span class="p">:</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
</code></pre></div>
<p>which tries to cover the methods a list-like object shall provide. This test, however, accepts objects that do not really behave like a list such as</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">FakeList</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">fakemethod</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">pass</span>
<span class="k">def</span> <span class="fm">__getattr__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="k">if</span> <span class="n">name</span> <span class="ow">in</span> <span class="p">[</span><span class="s1">'append'</span><span class="p">,</span> <span class="s1">'count'</span><span class="p">,</span> <span class="s1">'extend'</span><span class="p">,</span> <span class="s1">'index'</span><span class="p">,</span> <span class="s1">'insert'</span><span class="p">,</span> <span class="o">...</span><span class="p">]:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">fakemethod</span>
</code></pre></div>
<p>It is unlikely that you will write such a class, but this shows you one of the potential pitfalls of the previous test, which is wrong because it tries to rely on the structure instead of testing the behaviour. The temptation to rely on <code>isinstance()</code> is big</p>
<div class="highlight"><pre><span></span><code><span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">someobj</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span>
<span class="o">...</span>
</code></pre></div>
<p>If possible this approach is worse than before, since it tests the exact type. Even if <code>isinstance()</code> and <code>issubclass()</code> are smart enough to walk through the hierarchy of parent classes, this excludes evary class that behaves like a list does not inherit from it.</p>
<h2 id="back-to-delegation">Back to delegation<a class="headerlink" href="#back-to-delegation" title="Permanent link">¶</a></h2>
<p>The idea proposed in <a href="http://legacy.python.org/dev/peps/pep-3119/">PEP 3119</a> to face this problem is very elegant, and leverages the very nature of Python: that of being strongly based on delegation. The solution, implemented in Python 3 and backported to Python 2.7, changes the nature of the two <code>isinstance()</code> and <code>issubclass()</code> builtins. Now the first thing that <code>isinstance()</code> does is to call the <code>__instancecheck__()</code> method of the queried class, basically giving it the chance to answer the call with a different algorithm than the standard one. The same happens for <code>issubclass()</code>, which becomes <code>__subclasscheck__()</code>. So the following code</p>
<div class="highlight"><pre><span></span><code><span class="nb">issubclass</span><span class="p">(</span><span class="n">myclass</span><span class="p">,</span> <span class="n">someclass</span><span class="p">)</span>
</code></pre></div>
<p>does no more perform a pure external check of the relationship between <code>someclass</code> and <code>myclass</code>. The first thing that <code>issubclass()</code> does now is the following</p>
<div class="highlight"><pre><span></span><code><span class="n">someclass</span><span class="o">.</span><span class="fm">__subclasscheck__</span><span class="p">(</span><span class="n">myclass</span><span class="p">)</span>
</code></pre></div>
<p>This is very natural because, after all, <code>someclass</code> is the best source of judgement about being a subclass of itself.</p>
<h2 id="a-new-type-of-subclass">A new type of subclass<a class="headerlink" href="#a-new-type-of-subclass" title="Permanent link">¶</a></h2>
<p>With the introduction of delegation-based instance and subclass checks, Python provides a new type of subclass, and thus a new way to relate classes together. Now a subclass may be a real subclass, obtained using inheritance</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">ChildClass</span><span class="p">(</span><span class="n">ParentClass</span><span class="p">):</span>
<span class="k">pass</span>
</code></pre></div>
<p>or can be a <em>virtual</em> subclass, obtained through registration</p>
<div class="highlight"><pre><span></span><code><span class="n">ParentClass</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">ChildClass</span><span class="p">)</span>
</code></pre></div>
<p>The difference between a real and a virtual subclass is very simple: a real subclass knows its relationship with the parent class through its <code>__bases__</code> attribute, and can thus implicitly delegate the resolution of missing methods. A virtual subclass knows nothing about the class that registered it, and nowhere in the subclass will you find something that links it to the parent class. Thus, a virtual parent class is useful only as a categorization.</p>
<h2 id="abstract-base-classes">Abstract Base Classes<a class="headerlink" href="#abstract-base-classes" title="Permanent link">¶</a></h2>
<p>Classes that can register other classes, thus becoming virtual parents of those, are called in Python Abstract Base Classes, or ABCs.</p>
<p>The name of this new language element is important. ABCs are first of all classes, just like any other class you can create in Python, and they can be subclassed in the usual way to create taxonomies. They are also meant to be base classes, that is classes that represent fundamental behaviours or categories. Last, they are abstract. This has a very precise meaning in Python and will be the subject of the last part of this post.</p>
<p>The classes provided by the <code>collections</code> module are Abstract Base Classes, and they set themselves as virtual parents of some base types in the same module. If you check the <code>_collections_abc.py</code> file in your Python 3 installation (for example in <code>/usr/lib/python3.4/_collections_abc.py</code>) you will find code like this</p>
<div class="highlight"><pre><span></span><code><span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="n">Sequence</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="nb">tuple</span><span class="p">)</span>
<span class="n">Sequence</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="nb">str</span><span class="p">)</span>
<span class="n">Sequence</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="nb">range</span><span class="p">)</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="n">MutableSequence</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="nb">list</span><span class="p">)</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
</code></pre></div>
<p>Where the <code>Sequence</code> and the <code>MutableSequence</code> ABCs register some built-in types of Python.</p>
<p>It is very important to understand that registering a class does not imply any form of check about methods or attributes. Registering is just the <em>promise</em> that a given behaviour is provided by the registered class.</p>
<p>To demonstrate this let me provide you a very simple example made using one of the <code>collections</code> classes</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">collections</span>
<span class="gp">>>> </span><span class="k">class</span> <span class="nc">MyClass</span><span class="p">():</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">...</span>
<span class="gp">>>> </span><span class="nb">issubclass</span><span class="p">(</span><span class="n">MyClass</span><span class="p">,</span> <span class="n">collections</span><span class="o">.</span><span class="n">Sequence</span><span class="p">)</span>
<span class="go">False</span>
<span class="gp">>>> </span><span class="n">collections</span><span class="o">.</span><span class="n">Sequence</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">MyClass</span><span class="p">)</span>
<span class="go"><class '__main__.MyClass'></span>
<span class="gp">>>> </span><span class="nb">issubclass</span><span class="p">(</span><span class="n">MyClass</span><span class="p">,</span> <span class="n">collections</span><span class="o">.</span><span class="n">Sequence</span><span class="p">)</span>
<span class="go">True</span>
<span class="gp">>>></span>
</code></pre></div>
<p>As you can see, the <code>MyClass</code> class is initially not recognized as a subclass of <code>collections.Sequence</code>, but after the registration <code>issubclass()</code> returns <code>True</code>, even if the class is still empty.</p>
<h2 id="how-to-create-abcs">How to create ABCs<a class="headerlink" href="#how-to-create-abcs" title="Permanent link">¶</a></h2>
<p>The example given by the official documentation is very simple and clear</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">abc</span> <span class="kn">import</span> <span class="n">ABCMeta</span>
<span class="k">class</span> <span class="nc">MyABC</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">ABCMeta</span><span class="p">):</span>
<span class="k">pass</span>
<span class="n">MyABC</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="nb">tuple</span><span class="p">)</span>
<span class="k">assert</span> <span class="nb">issubclass</span><span class="p">(</span><span class="nb">tuple</span><span class="p">,</span> <span class="n">MyABC</span><span class="p">)</span>
<span class="k">assert</span> <span class="nb">isinstance</span><span class="p">((),</span> <span class="n">MyABC</span><span class="p">)</span>
</code></pre></div>
<p>All you need to do is to create a class and use the <code>ABCMeta</code> metaclass provided by the <code>abc</code> module and you will obtain a class that has the <code>register()</code> method and a suitable implementation of <code>__subclasscheck__()</code> and <code>__instancecheck__()</code>. Checking again the <code>_collections_abc.py</code> file you can see that this is exactly the way the <code>collections</code> classes are implemented</p>
<div class="highlight"><pre><span></span><code><span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="k">class</span> <span class="nc">Hashable</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">ABCMeta</span><span class="p">):</span>
<span class="o">...</span>
<span class="k">class</span> <span class="nc">Iterable</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">ABCMeta</span><span class="p">):</span>
<span class="o">...</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
</code></pre></div>
<h2 id="are-you-scared-of-metaclasses">Are you scared of metaclasses?<a class="headerlink" href="#are-you-scared-of-metaclasses" title="Permanent link">¶</a></h2>
<p>Metaclasses are a strange topic in Python. Most of the times the advice given to the novice is "Don't use them", like they were an error of the language and something that shall be avoided.</p>
<p>I don't think so. As a matter of facts I definitely disagree with such position, for many reasons.</p>
<p>First of all, if you are programming in Python its better for you to understand everything Python provides you, both the good and the bad parts. Programming languages are tools, and you shall know their strengths and their limitations. Most of the times what we call "limitations" are just features that become a restraint just because we are not aware of them. The C language, for example, is not object-oriented. Is this a strength or a limitation? Python provides you a very powerful inspection mechanism. Is this a strength or a limitations? I could give countless other examples.</p>
<p>Second, powerful features are the one you should know better. After all, we use a language for the unique features it provides, not for the features it shares with other languages. I use Python because of its powerful polymorphism implementation, not because of loops and inheritance. Those are provided by Java and C++, too, for example. I write a device driver in C because of the closeness to the machine language and its speed, not because of the <code>int</code> and <code>float</code> types, which are provided by many other languages. So, since powerful features are what let the language do what others cannot, those are the ones you have to master.</p>
<p>Third, if a feature of a language is a design error, and it can be, you need to understand why it is an error, and how you can avoid using it. Before ES6, JavaScript had some issues with scopes, given by the behaviour of the <code>var</code> keyword. You cannot afford being ignorant about those limitations, otherwise your software will be buggy. So, while studying the <code>for</code> loop in JavaScript took me a couple of minutes (it is a C-like for loop, after all), I spent a lot of time in dealing with <code>var</code>, which is the dangerous button of the whole device.</p>
<p>Back to Python. Metaclasses are not a last-minute feature put into the language just for fun. They are the foundation of Python itself, and the relationship between <code>object</code> and <code>type</code> is something so beautiful that it is a pity that basically no one talks about it. So, plase stop complaining against metaclasses and telling people that they are dangerous or complex.</p>
<p>Metaclasses are part of the language. And they are not complex to understand.</p>
<h2 id="why-metaclasses-for-abcs">Why metaclasses for ABCs?<a class="headerlink" href="#why-metaclasses-for-abcs" title="Permanent link">¶</a></h2>
<p>If you program in Python you should be somehow familiar with classes and instances. You know that when you build an instance you use a class (like a blueprint) and that the class can put things into the instance. For example</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Class definition</span>
<span class="k">class</span> <span class="nc">Child</span><span class="p">():</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">answer</span> <span class="o">=</span> <span class="mi">42</span>
<span class="c1"># Link instance and class</span>
<span class="n">c</span> <span class="o">=</span> <span class="n">Child</span><span class="p">()</span>
<span class="c1"># Use the instance</span>
<span class="k">assert</span> <span class="n">c</span><span class="o">.</span><span class="n">answer</span> <span class="o">==</span> <span class="mi">42</span>
</code></pre></div>
<p>Now, when you build a class you use a metaclass (like a blueprint) and the metaclass can put things into the class.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Metaclass definition</span>
<span class="k">class</span> <span class="nc">NewType</span><span class="p">(</span><span class="nb">type</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">bases</span><span class="p">,</span> <span class="n">namespace</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">answer</span> <span class="o">=</span> <span class="mi">42</span>
<span class="c1"># Link class and metaclass</span>
<span class="k">class</span> <span class="nc">Child</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">NewType</span><span class="p">):</span> <span class="k">pass</span>
<span class="c1"># Use the class</span>
<span class="k">assert</span> <span class="n">Child</span><span class="o">.</span><span class="n">answer</span> <span class="o">==</span> <span class="mi">42</span>
</code></pre></div>
<p>Sounds complex? Not at all, in my opinion. If you check the two examples you will see that they are exactly the same thing, the first referring to the instance-class relationship, the second to the class-metaclass one.</p>
<p>This is all you need to understand metaclasses. When you build a class you need to put things into it, for example you need to put the <code>__getattribute__</code> or the <code>__new__()</code> methods. This is done by the metaclass, which is usually <code>type</code> for every class. Indeed, if you check the <code>__class__</code> attribute into a class you get exactly this</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="nb">int</span>
<span class="go"><class 'int'></span>
<span class="gp">>>> </span><span class="nb">int</span><span class="o">.</span><span class="vm">__class__</span>
<span class="go"><class 'type'></span>
<span class="gp">>>></span>
</code></pre></div>
<h2 id="metaclasses-and-mro">Metaclasses and MRO<a class="headerlink" href="#metaclasses-and-mro" title="Permanent link">¶</a></h2>
<p>A slightly advanced annotation: when I say that the metaclass <em>puts</em> the methods into the class I'm simplifying the whole thing. As a matter of fact, like a class provides methods to the instance at runtime through the <code>__class__</code> attribute and the MRO protocol, the metaclass provides methods for the class. Attributes, instead, are put inside the class by the <code>__new__</code> or <code>__init__</code> methods of the metaclass.</p>
<p>Let us review the MRO mechanism for instances and classes first. When you call a method on an instance Python automatically looks for that method in the instance first, then in the parent class and in every class in its hierarchy.</p>
<p>So in this example</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">GrandParent</span><span class="p">():</span> <span class="k">pass</span>
<span class="k">class</span> <span class="nc">Parent</span><span class="p">(</span><span class="n">GrandParent</span><span class="p">):</span> <span class="k">pass</span>
<span class="k">class</span> <span class="nc">Child</span><span class="p">(</span><span class="n">Parent</span><span class="p">):</span> <span class="k">pass</span>
</code></pre></div>
<p>calling the method <code>get_name()</code> on an instance of <code>Child</code> will look for it first into the <code>Child</code> class, then into <code>Parent</code> and <code>GrandParent</code>, in this order. Finally, it will check <code>object</code>.</p>
<p>What happens to the MRO when a class of this hierarchy defines a different metaclass? For example</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">NewType</span><span class="p">(</span><span class="nb">type</span><span class="p">):</span> <span class="k">pass</span>
<span class="k">class</span> <span class="nc">GrandParent</span><span class="p">():</span> <span class="k">pass</span>
<span class="k">class</span> <span class="nc">Parent</span><span class="p">(</span><span class="n">GrandParent</span><span class="p">):</span> <span class="k">pass</span>
<span class="k">class</span> <span class="nc">Child</span><span class="p">(</span><span class="n">Parent</span><span class="p">,</span> <span class="n">metaclass</span><span class="o">=</span><span class="n">NewType</span><span class="p">):</span> <span class="k">pass</span>
</code></pre></div>
<p>In this case everything works as usual, but after checking <code>object</code> the MRO will also check the <code>NewType</code> metaclass (and its ancestors).</p>
<p>So, metaclasses can act as mixins, and they are queried only at the end of the usual MRO. This is exactly what happens using multiple inheritance if <code>NewType</code> were a standard parent class that does not have <code>Parent</code> or <code>GrandParent</code> as ancestors.</p>
<p>Metaclasses are not part of the MRO however, since the MRO just deals with standard inheritance. If you check the MRO of the <code>Child</code> class, you will see that the metaclass is not included</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">Child</span><span class="o">.</span><span class="n">mro</span><span class="p">()</span>
<span class="go">[<class '__main__.Child'>, <class '__main__.Parent'>, <class '__main__.GrandParent'>, <class 'object'>]</span>
<span class="gp">>>> </span><span class="n">Child</span><span class="o">.</span><span class="vm">__class__</span>
<span class="go"><class '__main__.NewType'></span>
<span class="gp">>>> </span>
</code></pre></div>
<h2 id="abstract-methods">Abstract methods<a class="headerlink" href="#abstract-methods" title="Permanent link">¶</a></h2>
<p>Why are ABC called <code>abstract</code>? ABCs can be instantiated, so they are after all not pure interfaces (like Java ones, for example)</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">abc</span>
<span class="gp">>>> </span><span class="k">class</span> <span class="nc">MyABC</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">abc</span><span class="o">.</span><span class="n">ABCMeta</span><span class="p">):</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="n">m</span> <span class="o">=</span> <span class="n">MyABC</span><span class="p">()</span>
<span class="gp">>>> </span>
</code></pre></div>
<p>They may however define some methods as <em>abstract</em>, using the <code>abc.abstractmethod</code> decorator, which prevents the class from being instantiated if the method is not implemented. Let me give you a simple example: I define an Abstract Base Class with and abstract method</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">MyABC</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">abc</span><span class="o">.</span><span class="n">ABCMeta</span><span class="p">):</span>
<span class="nd">@abc</span><span class="o">.</span><span class="n">abstractmethod</span>
<span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">pass</span>
</code></pre></div>
<p>and try to instantiate it. Python complains</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">m</span> <span class="o">=</span> <span class="n">MyABC</span><span class="p">()</span>
<span class="gt">Traceback (most recent call last):</span>
File <span class="nb">"<stdin>"</span>, line <span class="m">1</span>, in <span class="n"><module></span>
<span class="gr">TypeError</span>: <span class="n">Can't instantiate abstract class MyABC with abstract methods get</span>
</code></pre></div>
<p>I am forced to create a new class that inherits from <code>MyABC</code> and implements the method</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">Concrete</span><span class="p">(</span><span class="n">MyABC</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="mi">1</span>
</code></pre></div>
<p>Now I can instantiate the class.</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">c</span> <span class="o">=</span> <span class="n">Concrete</span><span class="p">()</span>
<span class="gp">>>> </span>
</code></pre></div>
<p>Check the official documentation of Abstract Base Classes for a complete description of <code>@abstractmethod</code> and <code>@abstractproperty</code>.</p>
<h2 id="what-about-the-behaviour">What about the behaviour?<a class="headerlink" href="#what-about-the-behaviour" title="Permanent link">¶</a></h2>
<p>So, what happened to the Python tenet "Check the behaviour and not the structure"? With collections, after all, we dropped the EAFP protocol, going back to a Look Before You Leap approach. Are we going against the philosophy at the very base of the language?</p>
<p>It is very interesting to see what Guido van Rossum, creator of the Python language, says about this in <a href="http://legacy.python.org/dev/peps/pep-3119/">PEP 3119</a>: <em>Invocation means interacting with an object by invoking its methods. Usually this is combined with polymorphism, so that invoking a given method may run different code depending on the type of an object. Inspection means the ability for external code (outside of the object's methods) to examine the type or properties of that object, and make decisions on how to treat that object based on that information. [...] In classical OOP theory, invocation is the preferred usage pattern, and inspection is actively discouraged, being considered a relic of an earlier, procedural programming style. However, in practice this view is simply too dogmatic and inflexible, and leads to a kind of design rigidity that is very much at odds with the dynamic nature of a language like Python.</em></p>
<p>So the point is that forcing the use of a pure polymorphic approach sometimes can lead to solutions that are too complex or even incorrect. The key words here, in my opinion, are "dogmatic", "inflexible", and "rigidity", opposed to "dynamic nature". I really like this flexibility in a language and in its author.</p>
<p>Writing <code>if isinstance(obj, collections.Sequence)</code> is not EAFP, neither is any conditional test you may write. Nevertheless, no one would replace conditional tests with a pure EAFP approach, simply because sometimes those tests are more readable. This is the exact purpose of collections in Python and ABCs in general: to allow parts of the code to be simpler.</p>
<h2 id="final-words">Final words<a class="headerlink" href="#final-words" title="Permanent link">¶</a></h2>
<p>I hope this post helped you understand that Abstract Base Classes, and in particular the standard collections, are useful and easy to understand. Metaclasses are also not that scary and dangerous, even if using them obviously requires some skill.</p>
<h2 id="sources">Sources<a class="headerlink" href="#sources" title="Permanent link">¶</a></h2>
<p>The official documentation of the <code>abc</code> module is very well written. <a href="https://docs.python.org/3.5/library/abc.html">Here</a> you find the version for Python 3.5. I also suggest to read the original <a href="https://www.python.org/dev/peps/pep-3119">PEP 3119</a> and the related <a href="https://www.python.org/dev/peps/pep-3141">PEP 3141</a> for a deeper understanding of the topic.</p>
<h2 id="feedback">Feedback<a class="headerlink" href="#feedback" title="Permanent link">¶</a></h2>
<p>The <a href="https://github.com/TheDigitalCatOnline/blog_source/issues">GitHub issues</a> page is the best place to submit corrections.</p>Python Mocks: a gentle introduction - Part 12016-03-06T18:00:00+01:002019-02-27T23:30:00+00:00Leonardo Giordanitag:www.thedigitalcatonline.com,2016-03-06:/blog/2016/03/06/python-mocks-a-gentle-introduction-part-1/<p>An introduction to mocks in Python: basic concepts and how to use them for tests</p><p>As already stressed in the two introductory posts on TDD (you can find them <a href="/categories/tdd/">here</a>) testing requires to write some code that uses the functions and objects you are going to develop. This means that you need to isolate a given (external) function that is part of your public API and demonstrate that it works with standard inputs and in edge cases.</p>
<p>For example, if you are going to develop an object that stores percentages (such as for example poll results), you should test the following conditions: the class can store a standard percentage such as 42%, the class shall give an error if you try to store a negative percentage, the class shall give an error if you store a percentage greater than 100%.</p>
<p>Tests shall be <em>idempotent</em> and <em>isolated</em>. Idempotent in mathematics and computer science identifies a process that can be run multiple times without changing the status of the system. Isolated means that a test shall not change its behaviour depending on previous executions of itself, nor depend on the previous execution (or missing execution) of other tests.</p>
<p>Such restrictions, which guarantee that your tests are not passing due to a temporary configuration of the system or the order in which they are run, can raise big issues when dealing with external libraries and systems, or with intrinsically mutable concepts such as time. In the testing discipline, such issues are mostly faced using mocks, that is objects that pretend to be other objects.</p>
<p>In this series of posts I am going to review the Python <code>mock</code> library and exemplify its use. I will not cover everything you may do with mock, obviously, but hopefully I'll give you the information you need to start using this powerful library.</p>
<h2 id="installation">Installation<a class="headerlink" href="#installation" title="Permanent link">¶</a></h2>
<p>First of all, <code>mock</code> is a Python library which development started around 2008. It was selected to be included in the standard library as of Python 3.3, which however does not prevent you to use other libraries if you prefer.</p>
<p>Python 3 users, thus, are not required to take any step, while for Python 2 projects you are still required to issue a <code>pip install mock</code> to install it into the system or the current virtualenv.</p>
<p>You may find the official documentation <a href="https://docs.python.org/dev/library/unittest.mock.html">here</a>. It is very detailed, and as always I strongly recommend taking your time to run through it.</p>
<h2 id="basic-concepts">Basic concepts<a class="headerlink" href="#basic-concepts" title="Permanent link">¶</a></h2>
<p>A mock, in the testing lingo, is an object that simulates the behaviour of another (more complex) object. When you (unit)test an object of your library you need sometimes to access other systems your object want to connect to, but you do not really want to be forced to run them, for several reasons.</p>
<p>The first one is that connecting with external systems means having a complex testing environment, that is you are dropping the isolation requirement of you tests. If your object wants to connect with a website, for example, you are forced to have a running Internet connection, and if the remote website is down you cannot test your library.</p>
<p>The second reason is that the setup of an external system is usually slow in comparison with the speed of unit tests. We expect to run hundred of tests in seconds, and if we have to fetch information from a remote server for each of them the time easily increases by several orders of magnitude. Remember: having slow tests means that you cannot run them while you develop, which in turn means that you will not really use them for TDD.</p>
<p>The third reason is more subtle, and has to to with the mutable nature of an external system, thus I'll postpone the discussion of this issue for the moment.</p>
<p>Let us try and work with a mock in Python and see what it can do. First of all fire up a Python shell or a <a href="http://jupyter.org">Jupyter Notebook</a> and import the library </p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">unittest</span> <span class="kn">import</span> <span class="n">mock</span>
</code></pre></div>
<p>If you are using Python 2 you have to install it and use</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">mock</span>
</code></pre></div>
<p>The main object that the library provides is <code>Mock</code> and you can instantiate it without any argument</p>
<div class="highlight"><pre><span></span><code><span class="n">m</span> <span class="o">=</span> <span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">()</span>
</code></pre></div>
<p>This object has the peculiar property of creating methods and attributes on the fly when you require them. Let us first look inside the object to take a glance of what it provides</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="nb">dir</span><span class="p">(</span><span class="n">m</span><span class="p">)</span>
<span class="go">['assert_any_call', 'assert_called_once_with', 'assert_called_with', 'assert_has_calls', 'attach_mock', 'call_args', 'call_args_list', 'call_count', 'called', 'configure_mock', 'method_calls', 'mock_add_spec', 'mock_calls', 'reset_mock', 'return_value', 'side_effect']</span>
</code></pre></div>
<p>As you can see there are some methods which are already defined into the <code>Mock</code> object. Let us read a non-existent attribute</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span>
<span class="go"><Mock name='mock.some_attribute' id='140222043808432'></span>
<span class="gp">>>> </span><span class="nb">dir</span><span class="p">(</span><span class="n">m</span><span class="p">)</span>
<span class="go">['assert_any_call', 'assert_called_once_with', 'assert_called_with', 'assert_has_calls', 'attach_mock', 'call_args', 'call_args_list', 'call_count', 'called', 'configure_mock', 'method_calls', 'mock_add_spec', 'mock_calls', 'reset_mock', 'return_value', 'side_effect', 'some_attribute']</span>
</code></pre></div>
<p>Well, as you can see this class is somehow different from what you are accustomed to. First of all its instances do not raise an <code>AttributeError</code> when asked for a non-existent attribute, but they happily return another instance of <code>Mock</code> itself. Second, the attribute you tried to access has now been created inside the object and accessing it returns the same mock object as before.</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span>
<span class="go"><Mock name='mock.some_attribute' id='140222043808432'></span>
</code></pre></div>
<p>Mock objects are callables, which means that they may act both as attributes and as methods. If you try to call the mock it just returns you another mock with a name that includes parentheses to signal its callable nature</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="p">()</span>
<span class="go"><Mock name='mock.some_attribute()' id='140247621475856'></span>
</code></pre></div>
<p>As you can understand, such objects are the perfect tool to mimic other objects or systems, since they may expose any API without raising exceptions. To use them in tests, however, we need them to behave just like the original, which implies returning sensible values or performing operations.</p>
<h2 id="return-value">Return value<a class="headerlink" href="#return-value" title="Permanent link">¶</a></h2>
<p>The simplest thing a mock can do for you is to return a given value every time you call it. This is configured setting the <code>return_value</code> attribute of a mock object</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some</span> <span class="n">attribute</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="mi">42</span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some</span> <span class="n">attribute</span><span class="p">()</span>
<span class="go">42</span>
</code></pre></div>
<p>Now the object does not return a mock object any more, instead it just returns the static value stored in the <code>return_value</code> attribute. Obviously you can also store a callable such as a function or an object, and the method will return it, but it will not run it. Let me give you an example</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="k">def</span> <span class="nf">print_answer</span><span class="p">():</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="s2">"42"</span><span class="p">)</span>
<span class="gp">... </span>
<span class="gp">>>> </span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="n">print_answer</span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="p">()</span>
<span class="go"><function print_answer at 0x7f8df1e3f400></span>
</code></pre></div>
<p>As you can see calling <code>some_attribute()</code> just returns the value stored in <code>return_value</code>, that is the function itself. To return values that come from a function we have to use a slightly more complex attribute of mock objects called <code>side_effect</code>.</p>
<h2 id="side-effect">Side effect<a class="headerlink" href="#side-effect" title="Permanent link">¶</a></h2>
<p>The <code>side_effect</code> parameter of mock objects is a very powerful tool. It accepts three different flavours of objects, callables, iterables, and exceptions, and changes its behaviour accordingly.</p>
<p>If you pass an exception the mock will raise it</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="o">.</span><span class="n">side_effect</span> <span class="o">=</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'A custom value error'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="p">()</span>
<span class="gt">Traceback (most recent call last):</span>
File <span class="nb">"<stdin>"</span>, line <span class="m">1</span>, in <span class="n"><module></span>
File <span class="nb">"/usr/lib/python3.4/unittest/mock.py"</span>, line <span class="m">902</span>, in <span class="n">__call__</span>
<span class="w"> </span><span class="k">return</span> <span class="n">_mock_self</span><span class="o">.</span><span class="n">_mock_call</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
File <span class="nb">"/usr/lib/python3.4/unittest/mock.py"</span>, line <span class="m">958</span>, in <span class="n">_mock_call</span>
<span class="w"> </span><span class="k">raise</span> <span class="n">effect</span>
<span class="gr">ValueError</span>: <span class="n">A custom value error</span>
</code></pre></div>
<p>If you pass an iterable, such as for example a generator, or a plain list, tuple, or similar objects, the mock will yield the values of that iterable, i.e. return every value contained in the iterable on subsequent calls of the mock. Let me give you an example</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="o">.</span><span class="n">side_effect</span> <span class="o">=</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="p">()</span>
<span class="go">0</span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="p">()</span>
<span class="go">1</span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="p">()</span>
<span class="go">2</span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="p">()</span>
<span class="gt">Traceback (most recent call last):</span>
File <span class="nb">"<stdin>"</span>, line <span class="m">1</span>, in <span class="n"><module></span>
File <span class="nb">"/usr/lib/python3.4/unittest/mock.py"</span>, line <span class="m">902</span>, in <span class="n">__call__</span>
<span class="w"> </span><span class="k">return</span> <span class="n">_mock_self</span><span class="o">.</span><span class="n">_mock_call</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
File <span class="nb">"/usr/lib/python3.4/unittest/mock.py"</span>, line <span class="m">961</span>, in <span class="n">_mock_call</span>
<span class="w"> </span><span class="n">result</span> <span class="o">=</span> <span class="nb">next</span><span class="p">(</span><span class="n">effect</span><span class="p">)</span>
<span class="gr">StopIteration</span>
</code></pre></div>
<p>As promised, the mock just returns every object found in the iterable (in this case a <code>range</code> object) once at a time until the generator is exhausted. According to the iterator protocol (see <a href="https://www.thedigitalcatonline.com/blog/2013/03/25/python-generators-from-iterators-to-cooperative-multitasking/">this post</a>) once every item has been returned the object raises the <code>StopIteration</code> exception, which means that you can correctly use it in a loop.</p>
<p>The last and perhaps most used case is that of passing a callable to <code>side_effect</code>, which shamelessly executes it with its own same parameters. This is very powerful, especially if you stop thinking about "functions" and start considering "callables". Indeed, <code>side_effect</code> also accepts a class and calls it, that is it can instantiate objects. Let us consider a simple example with a function without arguments</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="k">def</span> <span class="nf">print_answer</span><span class="p">():</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="s2">"42"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="o">.</span><span class="n">side_effect</span> <span class="o">=</span> <span class="n">print_answer</span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="p">()</span>
<span class="go">42</span>
</code></pre></div>
<p>A slightly more complex example: a function with arguments</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="k">def</span> <span class="nf">print_number</span><span class="p">(</span><span class="n">num</span><span class="p">):</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="s2">"Number:"</span><span class="p">,</span> <span class="n">num</span><span class="p">)</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="o">.</span><span class="n">side_effect</span> <span class="o">=</span> <span class="n">print_number</span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="o">.</span><span class="n">side_effect</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
<span class="go">Number: 5</span>
</code></pre></div>
<p>And finally an example with a class</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="k">class</span> <span class="nc">Number</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="gp">... </span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="gp">... </span> <span class="bp">self</span><span class="o">.</span><span class="n">_value</span> <span class="o">=</span> <span class="n">value</span>
<span class="gp">... </span> <span class="k">def</span> <span class="nf">print_value</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="s2">"Value:"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_value</span><span class="p">)</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="o">.</span><span class="n">side_effect</span> <span class="o">=</span> <span class="n">Number</span>
<span class="gp">>>> </span><span class="n">n</span> <span class="o">=</span> <span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="o">.</span><span class="n">side_effect</span><span class="p">(</span><span class="mi">26</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">n</span>
<span class="go"><__main__.Number object at 0x7f8df1aa4470></span>
<span class="gp">>>> </span><span class="n">n</span><span class="o">.</span><span class="n">print_value</span><span class="p">()</span>
<span class="go">Value: 26</span>
</code></pre></div>
<h2 id="testing-with-mocks">Testing with mocks<a class="headerlink" href="#testing-with-mocks" title="Permanent link">¶</a></h2>
<p>Now we know how to build a mock and how to give it a static return value or make it call a callable object. It is time to see how to use a mock in a test and what facilities do mocks provide. I'm going to use <a href="http://pytest.org">pytest</a> as a testing framework. You can find a quick introduction to pytest and TDD <a href="/categories/tdd/">here</a>).</p>
<h3 id="setup">Setup<a class="headerlink" href="#setup" title="Permanent link">¶</a></h3>
<p>If you want to quickly setup a pytest playground you may execute this code in a terminal (you need to have Python 3 and virtualenv installed in your system) </p>
<div class="highlight"><pre><span></span><code>mkdir<span class="w"> </span>mockplayground
<span class="nb">cd</span><span class="w"> </span>mockplayground
virtualenv<span class="w"> </span>venv3<span class="w"> </span>-p<span class="w"> </span>python3
<span class="nb">source</span><span class="w"> </span>venv3/bin/activate
pip<span class="w"> </span>install<span class="w"> </span>--upgrade<span class="w"> </span>pip
pip<span class="w"> </span>install<span class="w"> </span>pytest
<span class="nb">echo</span><span class="w"> </span><span class="s2">"[pytest]"</span><span class="w"> </span>>><span class="w"> </span>pytest.ini
<span class="nb">echo</span><span class="w"> </span><span class="s2">"norecursedirs=venv*"</span><span class="w"> </span>>><span class="w"> </span>pytest.ini
mkdir<span class="w"> </span>tests
touch<span class="w"> </span>myobj.py
touch<span class="w"> </span>tests/test_mock.py
<span class="nv">PYTHONPATH</span><span class="o">=</span><span class="s2">"."</span><span class="w"> </span>py.test
</code></pre></div>
<p>The <code>PYTHONPATH</code> environment variable is an easy way to avoid having to setup a whole Python project to just test some simple code.</p>
<h3 id="the-three-test-types">The three test types<a class="headerlink" href="#the-three-test-types" title="Permanent link">¶</a></h3>
<p>According to Sandy Metz we need to test only three types of messages (calls) between objects:</p>
<ul>
<li>Incoming queries (assertion on result)</li>
<li>Incoming commands (assertion on direct public side effects)</li>
<li>Outgoing commands (expectation on call and arguments)</li>
</ul>
<p>You can see the original talk <a href="https://www.youtube.com/watch?v=URSWYvyc42M">here</a> or read the slides <a href="https://speakerdeck.com/skmetz/magic-tricks-of-testing-railsconf">here</a>. The final table is shown in slide number 176.</p>
<p>As you can see when dealing with external objects we are only interested in knowing IF a method was called and WHAT PARAMETERS the caller passed to the object. We are not testing if the remote object returns the correct result, this is faked by the mock, which indeed returns exactly the result we need.</p>
<p>So the purpose of the methods provided by mock objects is to allow us to check what methods we called on the mock itself and what parameters we used in the call.</p>
<h3 id="asserting-calls">Asserting calls<a class="headerlink" href="#asserting-calls" title="Permanent link">¶</a></h3>
<p>To show how to use Python mocks in testing I will follow the TDD methodology, writing tests first and then writing the code that makes the tests pass. In this post I want to give you a simple overview of the mock objects, so I will not implement a real world use case, and the code will be very trivial. In the second part of this series I will test and implement a real class, in order to show some more interesting use cases.</p>
<p>The first thing we are usually interested in when dealing with an external object is to know that a given method has been called on it. Python mocks provide the <code>assert_called_with()</code> method to check this condition.</p>
<p>The use case we are going to test is the following. <em>We instantiate the <code>myobj.MyObj</code> class, which requires an external object. The class shall call the <code>connect()</code> method of the external object without any parameter.</em></p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">unittest</span> <span class="kn">import</span> <span class="n">mock</span>
<span class="kn">import</span> <span class="nn">myobj</span>
<span class="k">def</span> <span class="nf">test_instantiation</span><span class="p">():</span>
<span class="n">external_obj</span> <span class="o">=</span> <span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">()</span>
<span class="n">myobj</span><span class="o">.</span><span class="n">MyObj</span><span class="p">(</span><span class="n">external_obj</span><span class="p">)</span>
<span class="n">external_obj</span><span class="o">.</span><span class="n">connect</span><span class="o">.</span><span class="n">assert_called_with</span><span class="p">()</span>
</code></pre></div>
<p>The <code>myobj.MyObj</code> class, in this simple example, needs to connect to an external object, for example a remote repository or a database. The only thing we need to know for testing purposes is if the class called the <code>connect()</code> method of the external object without any parameter.</p>
<p>So the first thing we do in this test is to instantiate the mock object. This is a fake version of the external object, and its only purpose is to accept calls from the <code>MyObj</code> object under test and return sensible values. Then we instantiate the <code>MyObj</code> class passing the external object. We expect the class to call the <code>connect()</code> method so we express this expectation calling <code>external_obj.connect.assert_called_with()</code>.</p>
<p>What happens behind the scenes? The <code>MyObj</code> class receives the external object and somewhere is its initialization process calls the <code>connect()</code> method of the mock object and this creates the method itself as a mock object. This new mock records the parameters used to call it and the subsequent call to <code>assert_called_with()</code> checks that the method was called and that no parameters were passed.</p>
<p>Running pytest the test obviously fails.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nv">PYTHONPATH</span><span class="o">=</span><span class="s2">"."</span><span class="w"> </span>py.test
<span class="o">==========================================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">==========================================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.4.3+,<span class="w"> </span>pytest-2.9.0,<span class="w"> </span>py-1.4.31,<span class="w"> </span>pluggy-0.3.1
rootdir:<span class="w"> </span>/home/leo/devel/mockplayground,<span class="w"> </span>inifile:<span class="w"> </span>pytest.ini
collected<span class="w"> </span><span class="m">1</span><span class="w"> </span>items<span class="w"> </span>
tests/test_mock.py<span class="w"> </span><span class="nv">F</span>
<span class="o">===============================================</span><span class="w"> </span><span class="nv">FAILURES</span><span class="w"> </span><span class="o">================================================</span>
___________________________________________<span class="w"> </span>test_instantiation<span class="w"> </span>__________________________________________
<span class="w"> </span>def<span class="w"> </span>test_instantiation<span class="o">()</span>:
<span class="w"> </span><span class="nv">external_obj</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>mock.Mock<span class="o">()</span>
><span class="w"> </span>myobj.MyObj<span class="o">(</span>external_obj<span class="o">)</span>
E<span class="w"> </span>AttributeError:<span class="w"> </span><span class="s1">'module'</span><span class="w"> </span>object<span class="w"> </span>has<span class="w"> </span>no<span class="w"> </span>attribute<span class="w"> </span><span class="s1">'MyObj'</span>
tests/test_mock.py:6:<span class="w"> </span><span class="nv">AttributeError</span>
<span class="o">=======================================</span><span class="w"> </span><span class="m">1</span><span class="w"> </span>failed<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.03<span class="w"> </span><span class="nv">seconds</span><span class="w"> </span><span class="o">========================================</span>
$
</code></pre></div>
<p>Putting this code in <code>myobj.py</code> is enough to make the test pass</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">MyObj</span><span class="p">():</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">repo</span><span class="p">):</span>
<span class="n">repo</span><span class="o">.</span><span class="n">connect</span><span class="p">()</span>
</code></pre></div>
<p>As you can see, the <code>__init__()</code> method actually calls <code>repo.connect()</code>, where <code>repo</code> is expected to be a full-featured external object that provides a given API. In this case (for the moment) the API is just its <code>connect()</code> method. Calling <code>repo.connect()</code> when <code>repo</code> is a mock object silently creates the method as a mock object, as shown before.</p>
<p>The <code>assert_called_with()</code> method also allows us to check the parameters we passed when calling. To show this let us pretend that we expect the <code>MyObj.setup()</code> method to call <code>setup(cache=True, max_connections=256)</code> on the external object. As you can see we pass a couple of arguments (namely <code>cache</code> and <code>max_connections</code>) to the called method, and we want to be sure that the call was exactly in this form. The new test is thus</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">test_setup</span><span class="p">():</span>
<span class="n">external_obj</span> <span class="o">=</span> <span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">()</span>
<span class="n">obj</span> <span class="o">=</span> <span class="n">myobj</span><span class="o">.</span><span class="n">MyObj</span><span class="p">(</span><span class="n">external_obj</span><span class="p">)</span>
<span class="n">obj</span><span class="o">.</span><span class="n">setup</span><span class="p">()</span>
<span class="n">external_obj</span><span class="o">.</span><span class="n">setup</span><span class="o">.</span><span class="n">assert_called_with</span><span class="p">(</span><span class="n">cache</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">max_connections</span><span class="o">=</span><span class="mi">256</span><span class="p">)</span>
</code></pre></div>
<p>As usual the first run fails. Be sure to check this, it is part of the TDD methodology. You must have a test that DOES NOT PASS, then write some code that make it pass.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nv">PYTHONPATH</span><span class="o">=</span><span class="s2">"."</span><span class="w"> </span>py.test
<span class="o">==========================================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">==========================================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.4.3+,<span class="w"> </span>pytest-2.9.0,<span class="w"> </span>py-1.4.31,<span class="w"> </span>pluggy-0.3.1
rootdir:<span class="w"> </span>/home/leo/devel/mockplayground,<span class="w"> </span>inifile:<span class="w"> </span>pytest.ini
collected<span class="w"> </span><span class="m">2</span><span class="w"> </span>items<span class="w"> </span>
tests/test_mock.py<span class="w"> </span>.F
<span class="o">===============================================</span><span class="w"> </span><span class="nv">FAILURES</span><span class="w"> </span><span class="o">================================================</span>
______________________________________________<span class="w"> </span>test_setup<span class="w"> </span>_______________________________________________
<span class="w"> </span>def<span class="w"> </span>test_setup<span class="o">()</span>:
<span class="w"> </span><span class="nv">external_obj</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>mock.Mock<span class="o">()</span>
<span class="w"> </span><span class="nv">obj</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>myobj.MyObj<span class="o">(</span>external_obj<span class="o">)</span>
><span class="w"> </span>obj.setup<span class="o">()</span>
E<span class="w"> </span>AttributeError:<span class="w"> </span><span class="s1">'MyObj'</span><span class="w"> </span>object<span class="w"> </span>has<span class="w"> </span>no<span class="w"> </span>attribute<span class="w"> </span><span class="s1">'setup'</span>
tests/test_mock.py:14:<span class="w"> </span><span class="nv">AttributeError</span>
<span class="o">==================================</span><span class="w"> </span><span class="m">1</span><span class="w"> </span>failed,<span class="w"> </span><span class="m">1</span><span class="w"> </span>passed<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.03<span class="w"> </span><span class="nv">seconds</span><span class="w"> </span><span class="o">===================================</span>
$
</code></pre></div>
<p>To show you what type of check the mock object provides let me implement a partially correct solution</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">MyObj</span><span class="p">():</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">repo</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_repo</span> <span class="o">=</span> <span class="n">repo</span>
<span class="n">repo</span><span class="o">.</span><span class="n">connect</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">setup</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_repo</span><span class="o">.</span><span class="n">setup</span><span class="p">(</span><span class="n">cache</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</code></pre></div>
<p>As you can see the external object has been stored in <code>self._repo</code> and the call to <code>self._repo.setup()</code> is not exactly what the test expects, lacking the <code>max_connections</code> parameter. Running pytest we obtain the following result (I removed most of the pytest output)</p>
<div class="highlight"><pre><span></span><code>E<span class="w"> </span>AssertionError:<span class="w"> </span>Expected<span class="w"> </span>call:<span class="w"> </span>setup<span class="o">(</span><span class="nv">cache</span><span class="o">=</span>True,<span class="w"> </span><span class="nv">max_connections</span><span class="o">=</span><span class="m">256</span><span class="o">)</span>
E<span class="w"> </span>Actual<span class="w"> </span>call:<span class="w"> </span>setup<span class="o">(</span><span class="nv">cache</span><span class="o">=</span>True<span class="o">)</span>
</code></pre></div>
<p>and you see that the error message is very clear about what we expected and what happened in our code.</p>
<p>As you can read in the official documentation, the <code>Mock</code> object also provides the following methods and attributes: <code>assert_called_once_with</code>, <code>assert_any_call</code>, <code>assert_has_calls</code>, <code>assert_not_called</code>, <code>called</code>, <code>call_count</code>. Each of them explores a different aspect of the mock behaviour concerning calls, make sure to check their description and the examples provided along.</p>
<h2 id="final-words">Final words<a class="headerlink" href="#final-words" title="Permanent link">¶</a></h2>
<p>In this first part of the series I described the behaviour of mock objects and the methods they provide to simulate return values and to test calls. They are a very powerful tool that allows you to avoid creating complex and slow tests that depend on external facilities to run, thus missing the main purpose of tests, which is that of <em>continuously</em> helping you to check your code.</p>
<p>In the next issue of the series I will explore the automatic creation of mock methods from a given object and the very important patching mechanism provided by the <code>patch</code> decorator and context manager.</p>
<h2 id="updates">Updates<a class="headerlink" href="#updates" title="Permanent link">¶</a></h2>
<p>2019-02-27: <a href="https://github.com/schneeemensch">schneeemensch</a> spotted a mistake in the code of the "Return value" section, where <code>m.some_attribute.side_effect()</code> was used instead of <code>m.some_attribute()</code>. Thanks!</p>
<h2 id="feedback">Feedback<a class="headerlink" href="#feedback" title="Permanent link">¶</a></h2>
<p>The <a href="https://github.com/TheDigitalCatOnline/blog_source/issues">GitHub issues</a> page is the best place to submit corrections.</p>