The Digital Cat - metaprogramminghttps://www.thedigitalcatonline.com/2020-04-26T00:00:00+01:00Adventures of a curious cat in the land of programmingObject-Oriented Programming (OOP) concepts in Python2020-04-26T00:00:00+01:002020-04-26T00:00:00+01:00Leonardo Giordanitag:www.thedigitalcatonline.com,2020-04-26:/blog/2020/04/26/object-oriented-programming-concepts-in-python/<p>A review of the main concepts that a beginner Python programmer should learn about OOP in Python</p><p>In this article I will review the main concepts a beginner Python programmer should learn about OOP in Python. The content is explored in detail in other posts on the blog that are linked at the end of each section.</p>
<h2 id="fundamentals">Fundamentals<a class="headerlink" href="#fundamentals" title="Permanent link">¶</a></h2>
<p>Object-oriented programming (OOP) is a paradigm, that is <em>a way to structure your code</em> that aims to make it <em>easy to maintain and modify</em>. Often OOP is considered opposed to functional programming (FP), but the truth is that both paradigms are useful and some modern languages implement both (Scala, for example). Python implements some features of the FP paradigm, but if you want to be a good Python programmer you have to understand OOP.</p>
<p>Moreover, since OOP is a conceptual paradigm, you have to learn how Python <em>implements</em> OOP, that is, the specific rules of the Python universe, which are not true in other programming languages.</p>
<p>The core concept you have to learn is that a paradigm like OOP has a single goal: that of making code <em>reusable</em> and <em>easy to change</em>. Keep it in mind, otherwise you will blindly learn rules without understanding, which means that you will not understand when you have to be strict and when you should bend them.</p>
<h2 id="three-pillars">Three pillars<a class="headerlink" href="#three-pillars" title="Permanent link">¶</a></h2>
<p>OOP has three main pillars:</p>
<ul>
<li><strong>Data encapsulation</strong> has been introduced to link data structures and functions working on them. Objects are created from <em>classes</em>, which promote the separation between private code (code used to manage the low-level details of the data structures) and public code (code available to other parts of the system).</li>
<li><strong>Delegation</strong> is the way OOP promotes <em>code reuse</em>. Through delegation, objects in a system can collaborate and avoid duplicating code. It can be implemented through <em>inheritance</em> (implicit delegation) or <em>composition</em> (explicit delegation).</li>
<li><strong>Polymorphism</strong> is how OOP promotes <em>algorithm reuse</em>. When polymorphism is correctly implemented, objects with completely different internal structures can be processed by the same algorithm and collaborate. This promotes <em>refactoring</em> and <em>code maintenance</em> allowing to replace old code with new implementations.</li>
</ul>
<h2 id="resources">Resources<a class="headerlink" href="#resources" title="Permanent link">¶</a></h2>
<p>To learn how Python implements these three main concepts you can read the following posts:</p>
<ul>
<li>Data encapsulation: <a href="https://www.thedigitalcatonline.com/blog/2014/08/20/python-3-oop-part-1-objects-and-types/">Object-Oriented Programming in Python 3 - Objects and types</a> and <a href="https://www.thedigitalcatonline.com/blog/2014/08/20/python-3-oop-part-2-classes-and-members/">
Object-Oriented Programming in Python 3 - Classes and members</a></li>
<li>Delegation: <a href="https://www.thedigitalcatonline.com/blog/2014/08/20/python-3-oop-part-3-delegation-composition-and-inheritance/">Object-Oriented Programming in Python 3 - Composition and inheritance</a></li>
<li>Polymorphism: <a href="https://www.thedigitalcatonline.com/blog/2014/08/21/python-3-oop-part-4-polymorphism/">Object-Oriented Programming in Python 3 - Polymorphism</a></li>
</ul>
<p>These are all part of the same series of posts, so you can read them in order to have a complete picture. The same concepts are also explained in the following videos that I published on YouTube:</p>
<ul>
<li>Video <a href="https://www.youtube.com/watch?v=1Jc41dIVOk8&list=PLWtCrYLGt7T3DUFPYdqrdEqzt-OCfBQ5O&index=2&t=0s">Object-oriented programming in Python - Part 1 - Basic concepts</a></li>
<li>Video <a href="https://www.youtube.com/watch?v=-O7OHmrQMfc&list=PLWtCrYLGt7T3DUFPYdqrdEqzt-OCfBQ5O&index=3&t=0s">Object-oriented programming in Python - Part 2 - Types and classes in Python</a></li>
<li>Video <a href="https://www.youtube.com/watch?v=_b18YfmEvTg&list=PLWtCrYLGt7T3DUFPYdqrdEqzt-OCfBQ5O&index=4&t=0s">Object-oriented programming in Python - Part 3 - How to create a class</a></li>
<li>Video <a href="https://www.youtube.com/watch?v=6ZPXR0Gj0GU&list=PLWtCrYLGt7T3DUFPYdqrdEqzt-OCfBQ5O&index=5&t=0s">Object-oriented programming in Python - Part 4 - Classes and instances</a></li>
<li>Video <a href="https://www.youtube.com/watch?v=FLwxJrwYHbo&list=PLWtCrYLGt7T3DUFPYdqrdEqzt-OCfBQ5O&index=6&t=0s">Object-oriented programming in Python - Part 5 - Delegation: inheritance and composition</a></li>
</ul>
<p>To get more comfortable with classes and delegation you can review them with these two simple articles:</p>
<ul>
<li><a href="https://www.thedigitalcatonline.com/blog/2015/01/12/accessing-attributes-in-python/">Accessing attributes in Python</a></li>
<li><a href="https://www.thedigitalcatonline.com/blog/2014/05/19/method-overriding-in-python/">Method overriding in Python</a></li>
</ul>
<p>The following posts are simple but very useful exercises of OOP design, and they also use test-driven development (TDD)</p>
<ul>
<li><a href="https://www.thedigitalcatonline.com/blog/2015/05/13/python-oop-tdd-example-part1/">A simple example of Python OOP development (with TDD) - Part 1</a> and <a href="https://www.thedigitalcatonline.com/blog/2015/09/10/python-oop-tdd-example-part2/">A simple example of Python OOP development (with TDD) - Part 2</a></li>
<li><a href="https://www.thedigitalcatonline.com/blog/2017/07/21/refactoring-with-test-in-python-a-practical-example/">Refactoring with tests in Python: a practical example</a></li>
</ul>
<p>Speaking of TDD, you should get used to mocks, that are extremely useful when it comes to test objects that work together:</p>
<ul>
<li><a href="https://www.thedigitalcatonline.com/blog/2016/03/06/python-mocks-a-gentle-introduction-part-1/">Python Mocks: a gentle introduction - Part 1</a> and <a href="https://www.thedigitalcatonline.com/blog/2016/09/27/python-mocks-a-gentle-introduction-part-2/">Python Mocks: a gentle introduction - Part 2</a></li>
</ul>
<p>Before you move on to more complex concepts you might want to have a look at what you can build with these basic concepts of OOP and TDD:</p>
<ul>
<li><a href="https://www.thedigitalcatonline.com/blog/2016/11/14/clean-architectures-in-python-a-step-by-step-example/">Clean architectures in Python: a step-by-step example</a> which evolved in my book "Clean Architectures in Python" available <a href="https://www.thedigitalcatbooks.com/pycabook-introduction/">online</a> or <a href="https://leanpub.com/clean-architectures-in-python">as PDF/ebook</a></li>
</ul>
<p>If you made it so far you are ready to face the big players! The following are concepts for advanced programmers, so be sure you are comfortable with the previous sections before venturing here. Don't be too scared, though. These concepts are advanced but not impossible, and if you want to get better at Python you should at least understand what they are.</p>
<ul>
<li>Metaclasses: <a href="https://www.thedigitalcatonline.com/blog/2014/09/01/python-3-oop-part-5-metaclasses/">Object-Oriented Programming in Python 3 - Metaclasses</a> and <a href="https://www.thedigitalcatonline.com/blog/2014/10/14/decorators-and-metaclasses/">Advanced use of Python decorators and metaclasses</a></li>
<li>Abstract Base Classes: <a href="https://www.thedigitalcatonline.com/blog/2014/09/04/python-3-oop-part-6-abstract-base-classes/">Object-Oriented Programming in Python 3 - Abstract Base Classes</a> and <a href="https://www.thedigitalcatonline.com/blog/2015/04/23/python-decorators-metaprogramming-with-style/">Python decorators: metaprogramming with style</a></li>
<li>Multiple inheritance: <a href="https://www.thedigitalcatonline.com/blog/2020/03/27/mixin-classes-in-python/">Multiple inheritance and mixin classes in Python</a></li>
</ul>
<h2 id="final-words">Final words<a class="headerlink" href="#final-words" title="Permanent link">¶</a></h2>
<p>Wow, this was a nice trip! Obviously, I don't think I covered all that can be said on OOP in Python, but I'm pretty sure that all these posts can boost your knowledge of this important paradigm and allow you to become a better Python programmer. I will update this post when I will publish new articles on OOP here on the blog.</p>
<p>If you found this useful, please consider sharing it with a friend! </p>
<h2 id="feedback">Feedback<a class="headerlink" href="#feedback" title="Permanent link">¶</a></h2>
<p>Feel free to reach me on <a href="https://twitter.com/thedigicat">Twitter</a> if you have questions. The <a href="https://github.com/TheDigitalCatOnline/blog_source/issues">GitHub issues</a> page is the best place to submit corrections.</p>
<p>Image by <a href="https://pixabay.com/users/LoggaWiggler-15/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=5236">LoggaWiggler</a> from <a href="https://pixabay.com/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=5236">Pixabay</a></p>Abstract Base Classes in Python2016-04-03T11:00:00+01:002016-04-03T11:00:00+01:00Leonardo Giordanitag:www.thedigitalcatonline.com,2016-04-03:/blog/2016/04/03/abstract-base-classes-in-python/<p>What are Abstract Base Classes in Python and why are they useful?</p><p>With the introduction of Abstract Base Classes, Python once again shows its nature of a very innovative and flexible language. It is interesting to see how such a remarkable feature has been introduced into the language by a pure Python module. This demonstrates that Python is built in a way that is very open to changes, thanks to its foundations in pure polymorphism based on delegation.</p>
<p>Many Python programmers overlooked Abstract Base Classes and the classes in the <code>collections</code> module, which are one of the simplest and useful applications of the concept. Sure enough, this is not a feature that you will use every day or that will change the way you are programming in Python. But neither is it something you shall discard before understanding what it brings into the language, and what sort of problems it can solve for you.</p>
<h2 id="eafp">EAFP<a class="headerlink" href="#eafp" title="Permanent link">¶</a></h2>
<p>Python is a dynamically-typed object-oriented language strongly based on delegation, so its approach to problems is intrinsically polymorphic. This means that Python deals mostly with the behaviour of objects and not with their structure. The well-known EAFP protocol (it's Easier to Ask Forgiveness than Permission) comes from this approach. This code</p>
<div class="highlight"><pre><span></span><code><span class="k">try</span><span class="p">:</span>
<span class="n">someobj</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="k">except</span> <span class="ne">TypeError</span><span class="p">:</span>
<span class="c1"># object is not subscriptable</span>
<span class="o">...</span>
</code></pre></div>
<p>does not check if the object is a <code>list</code> or a <code>dictionary</code> (both would allow the <code>[1]</code> notation), but if the object can be accessed by key (or index). When you accept a parameter in a function Python does not specify the type (leaving aside type hints) because you are not interested in accepting a given type or one of its derived types. You are interested in accepting something that provides the methods you will use.</p>
<p>The behaviour, in an object-oriented environment, is the run-time interface of the object. This is different from the static interface, which is the collection of the methods provided by the object. The run-time interface is the actual interface the object shows when it is used, and this encompasses the methods provided by its class, but also methods provided by parent classes, the metaclass and other entry points provided by <code>__getattr__</code>.</p>
<h2 id="complex-checks">Complex checks<a class="headerlink" href="#complex-checks" title="Permanent link">¶</a></h2>
<p>Sometimes, however, you need to perform complex checks, such as "it behaves like a list". How can you test this condition? You could test if the incoming object has some standard methods, but this is not only incomplete but also wrong. For example, I could write the following test</p>
<div class="highlight"><pre><span></span><code><span class="k">try</span><span class="p">:</span>
<span class="n">obj</span><span class="o">.</span><span class="n">append</span>
<span class="n">obj</span><span class="o">.</span><span class="n">count</span>
<span class="n">obj</span><span class="o">.</span><span class="n">extend</span>
<span class="n">obj</span><span class="o">.</span><span class="n">index</span>
<span class="n">obj</span><span class="o">.</span><span class="n">insert</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="k">except</span> <span class="ne">AttributeError</span><span class="p">:</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
</code></pre></div>
<p>which tries to cover the methods a list-like object shall provide. This test, however, accepts objects that do not really behave like a list such as</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">FakeList</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">fakemethod</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">pass</span>
<span class="k">def</span> <span class="fm">__getattr__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="k">if</span> <span class="n">name</span> <span class="ow">in</span> <span class="p">[</span><span class="s1">'append'</span><span class="p">,</span> <span class="s1">'count'</span><span class="p">,</span> <span class="s1">'extend'</span><span class="p">,</span> <span class="s1">'index'</span><span class="p">,</span> <span class="s1">'insert'</span><span class="p">,</span> <span class="o">...</span><span class="p">]:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">fakemethod</span>
</code></pre></div>
<p>It is unlikely that you will write such a class, but this shows you one of the potential pitfalls of the previous test, which is wrong because it tries to rely on the structure instead of testing the behaviour. The temptation to rely on <code>isinstance()</code> is big</p>
<div class="highlight"><pre><span></span><code><span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">someobj</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span>
<span class="o">...</span>
</code></pre></div>
<p>If possible this approach is worse than before, since it tests the exact type. Even if <code>isinstance()</code> and <code>issubclass()</code> are smart enough to walk through the hierarchy of parent classes, this excludes evary class that behaves like a list does not inherit from it.</p>
<h2 id="back-to-delegation">Back to delegation<a class="headerlink" href="#back-to-delegation" title="Permanent link">¶</a></h2>
<p>The idea proposed in <a href="http://legacy.python.org/dev/peps/pep-3119/">PEP 3119</a> to face this problem is very elegant, and leverages the very nature of Python: that of being strongly based on delegation. The solution, implemented in Python 3 and backported to Python 2.7, changes the nature of the two <code>isinstance()</code> and <code>issubclass()</code> builtins. Now the first thing that <code>isinstance()</code> does is to call the <code>__instancecheck__()</code> method of the queried class, basically giving it the chance to answer the call with a different algorithm than the standard one. The same happens for <code>issubclass()</code>, which becomes <code>__subclasscheck__()</code>. So the following code</p>
<div class="highlight"><pre><span></span><code><span class="nb">issubclass</span><span class="p">(</span><span class="n">myclass</span><span class="p">,</span> <span class="n">someclass</span><span class="p">)</span>
</code></pre></div>
<p>does no more perform a pure external check of the relationship between <code>someclass</code> and <code>myclass</code>. The first thing that <code>issubclass()</code> does now is the following</p>
<div class="highlight"><pre><span></span><code><span class="n">someclass</span><span class="o">.</span><span class="fm">__subclasscheck__</span><span class="p">(</span><span class="n">myclass</span><span class="p">)</span>
</code></pre></div>
<p>This is very natural because, after all, <code>someclass</code> is the best source of judgement about being a subclass of itself.</p>
<h2 id="a-new-type-of-subclass">A new type of subclass<a class="headerlink" href="#a-new-type-of-subclass" title="Permanent link">¶</a></h2>
<p>With the introduction of delegation-based instance and subclass checks, Python provides a new type of subclass, and thus a new way to relate classes together. Now a subclass may be a real subclass, obtained using inheritance</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">ChildClass</span><span class="p">(</span><span class="n">ParentClass</span><span class="p">):</span>
<span class="k">pass</span>
</code></pre></div>
<p>or can be a <em>virtual</em> subclass, obtained through registration</p>
<div class="highlight"><pre><span></span><code><span class="n">ParentClass</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">ChildClass</span><span class="p">)</span>
</code></pre></div>
<p>The difference between a real and a virtual subclass is very simple: a real subclass knows its relationship with the parent class through its <code>__bases__</code> attribute, and can thus implicitly delegate the resolution of missing methods. A virtual subclass knows nothing about the class that registered it, and nowhere in the subclass will you find something that links it to the parent class. Thus, a virtual parent class is useful only as a categorization.</p>
<h2 id="abstract-base-classes">Abstract Base Classes<a class="headerlink" href="#abstract-base-classes" title="Permanent link">¶</a></h2>
<p>Classes that can register other classes, thus becoming virtual parents of those, are called in Python Abstract Base Classes, or ABCs.</p>
<p>The name of this new language element is important. ABCs are first of all classes, just like any other class you can create in Python, and they can be subclassed in the usual way to create taxonomies. They are also meant to be base classes, that is classes that represent fundamental behaviours or categories. Last, they are abstract. This has a very precise meaning in Python and will be the subject of the last part of this post.</p>
<p>The classes provided by the <code>collections</code> module are Abstract Base Classes, and they set themselves as virtual parents of some base types in the same module. If you check the <code>_collections_abc.py</code> file in your Python 3 installation (for example in <code>/usr/lib/python3.4/_collections_abc.py</code>) you will find code like this</p>
<div class="highlight"><pre><span></span><code><span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="n">Sequence</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="nb">tuple</span><span class="p">)</span>
<span class="n">Sequence</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="nb">str</span><span class="p">)</span>
<span class="n">Sequence</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="nb">range</span><span class="p">)</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="n">MutableSequence</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="nb">list</span><span class="p">)</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
</code></pre></div>
<p>Where the <code>Sequence</code> and the <code>MutableSequence</code> ABCs register some built-in types of Python.</p>
<p>It is very important to understand that registering a class does not imply any form of check about methods or attributes. Registering is just the <em>promise</em> that a given behaviour is provided by the registered class.</p>
<p>To demonstrate this let me provide you a very simple example made using one of the <code>collections</code> classes</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">collections</span>
<span class="gp">>>> </span><span class="k">class</span> <span class="nc">MyClass</span><span class="p">():</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">...</span>
<span class="gp">>>> </span><span class="nb">issubclass</span><span class="p">(</span><span class="n">MyClass</span><span class="p">,</span> <span class="n">collections</span><span class="o">.</span><span class="n">Sequence</span><span class="p">)</span>
<span class="go">False</span>
<span class="gp">>>> </span><span class="n">collections</span><span class="o">.</span><span class="n">Sequence</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">MyClass</span><span class="p">)</span>
<span class="go"><class '__main__.MyClass'></span>
<span class="gp">>>> </span><span class="nb">issubclass</span><span class="p">(</span><span class="n">MyClass</span><span class="p">,</span> <span class="n">collections</span><span class="o">.</span><span class="n">Sequence</span><span class="p">)</span>
<span class="go">True</span>
<span class="gp">>>></span>
</code></pre></div>
<p>As you can see, the <code>MyClass</code> class is initially not recognized as a subclass of <code>collections.Sequence</code>, but after the registration <code>issubclass()</code> returns <code>True</code>, even if the class is still empty.</p>
<h2 id="how-to-create-abcs">How to create ABCs<a class="headerlink" href="#how-to-create-abcs" title="Permanent link">¶</a></h2>
<p>The example given by the official documentation is very simple and clear</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">abc</span> <span class="kn">import</span> <span class="n">ABCMeta</span>
<span class="k">class</span> <span class="nc">MyABC</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">ABCMeta</span><span class="p">):</span>
<span class="k">pass</span>
<span class="n">MyABC</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="nb">tuple</span><span class="p">)</span>
<span class="k">assert</span> <span class="nb">issubclass</span><span class="p">(</span><span class="nb">tuple</span><span class="p">,</span> <span class="n">MyABC</span><span class="p">)</span>
<span class="k">assert</span> <span class="nb">isinstance</span><span class="p">((),</span> <span class="n">MyABC</span><span class="p">)</span>
</code></pre></div>
<p>All you need to do is to create a class and use the <code>ABCMeta</code> metaclass provided by the <code>abc</code> module and you will obtain a class that has the <code>register()</code> method and a suitable implementation of <code>__subclasscheck__()</code> and <code>__instancecheck__()</code>. Checking again the <code>_collections_abc.py</code> file you can see that this is exactly the way the <code>collections</code> classes are implemented</p>
<div class="highlight"><pre><span></span><code><span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="k">class</span> <span class="nc">Hashable</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">ABCMeta</span><span class="p">):</span>
<span class="o">...</span>
<span class="k">class</span> <span class="nc">Iterable</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">ABCMeta</span><span class="p">):</span>
<span class="o">...</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
</code></pre></div>
<h2 id="are-you-scared-of-metaclasses">Are you scared of metaclasses?<a class="headerlink" href="#are-you-scared-of-metaclasses" title="Permanent link">¶</a></h2>
<p>Metaclasses are a strange topic in Python. Most of the times the advice given to the novice is "Don't use them", like they were an error of the language and something that shall be avoided.</p>
<p>I don't think so. As a matter of facts I definitely disagree with such position, for many reasons.</p>
<p>First of all, if you are programming in Python its better for you to understand everything Python provides you, both the good and the bad parts. Programming languages are tools, and you shall know their strengths and their limitations. Most of the times what we call "limitations" are just features that become a restraint just because we are not aware of them. The C language, for example, is not object-oriented. Is this a strength or a limitation? Python provides you a very powerful inspection mechanism. Is this a strength or a limitations? I could give countless other examples.</p>
<p>Second, powerful features are the one you should know better. After all, we use a language for the unique features it provides, not for the features it shares with other languages. I use Python because of its powerful polymorphism implementation, not because of loops and inheritance. Those are provided by Java and C++, too, for example. I write a device driver in C because of the closeness to the machine language and its speed, not because of the <code>int</code> and <code>float</code> types, which are provided by many other languages. So, since powerful features are what let the language do what others cannot, those are the ones you have to master.</p>
<p>Third, if a feature of a language is a design error, and it can be, you need to understand why it is an error, and how you can avoid using it. Before ES6, JavaScript had some issues with scopes, given by the behaviour of the <code>var</code> keyword. You cannot afford being ignorant about those limitations, otherwise your software will be buggy. So, while studying the <code>for</code> loop in JavaScript took me a couple of minutes (it is a C-like for loop, after all), I spent a lot of time in dealing with <code>var</code>, which is the dangerous button of the whole device.</p>
<p>Back to Python. Metaclasses are not a last-minute feature put into the language just for fun. They are the foundation of Python itself, and the relationship between <code>object</code> and <code>type</code> is something so beautiful that it is a pity that basically no one talks about it. So, plase stop complaining against metaclasses and telling people that they are dangerous or complex.</p>
<p>Metaclasses are part of the language. And they are not complex to understand.</p>
<h2 id="why-metaclasses-for-abcs">Why metaclasses for ABCs?<a class="headerlink" href="#why-metaclasses-for-abcs" title="Permanent link">¶</a></h2>
<p>If you program in Python you should be somehow familiar with classes and instances. You know that when you build an instance you use a class (like a blueprint) and that the class can put things into the instance. For example</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Class definition</span>
<span class="k">class</span> <span class="nc">Child</span><span class="p">():</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">answer</span> <span class="o">=</span> <span class="mi">42</span>
<span class="c1"># Link instance and class</span>
<span class="n">c</span> <span class="o">=</span> <span class="n">Child</span><span class="p">()</span>
<span class="c1"># Use the instance</span>
<span class="k">assert</span> <span class="n">c</span><span class="o">.</span><span class="n">answer</span> <span class="o">==</span> <span class="mi">42</span>
</code></pre></div>
<p>Now, when you build a class you use a metaclass (like a blueprint) and the metaclass can put things into the class.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Metaclass definition</span>
<span class="k">class</span> <span class="nc">NewType</span><span class="p">(</span><span class="nb">type</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">bases</span><span class="p">,</span> <span class="n">namespace</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">answer</span> <span class="o">=</span> <span class="mi">42</span>
<span class="c1"># Link class and metaclass</span>
<span class="k">class</span> <span class="nc">Child</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">NewType</span><span class="p">):</span> <span class="k">pass</span>
<span class="c1"># Use the class</span>
<span class="k">assert</span> <span class="n">Child</span><span class="o">.</span><span class="n">answer</span> <span class="o">==</span> <span class="mi">42</span>
</code></pre></div>
<p>Sounds complex? Not at all, in my opinion. If you check the two examples you will see that they are exactly the same thing, the first referring to the instance-class relationship, the second to the class-metaclass one.</p>
<p>This is all you need to understand metaclasses. When you build a class you need to put things into it, for example you need to put the <code>__getattribute__</code> or the <code>__new__()</code> methods. This is done by the metaclass, which is usually <code>type</code> for every class. Indeed, if you check the <code>__class__</code> attribute into a class you get exactly this</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="nb">int</span>
<span class="go"><class 'int'></span>
<span class="gp">>>> </span><span class="nb">int</span><span class="o">.</span><span class="vm">__class__</span>
<span class="go"><class 'type'></span>
<span class="gp">>>></span>
</code></pre></div>
<h2 id="metaclasses-and-mro">Metaclasses and MRO<a class="headerlink" href="#metaclasses-and-mro" title="Permanent link">¶</a></h2>
<p>A slightly advanced annotation: when I say that the metaclass <em>puts</em> the methods into the class I'm simplifying the whole thing. As a matter of fact, like a class provides methods to the instance at runtime through the <code>__class__</code> attribute and the MRO protocol, the metaclass provides methods for the class. Attributes, instead, are put inside the class by the <code>__new__</code> or <code>__init__</code> methods of the metaclass.</p>
<p>Let us review the MRO mechanism for instances and classes first. When you call a method on an instance Python automatically looks for that method in the instance first, then in the parent class and in every class in its hierarchy.</p>
<p>So in this example</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">GrandParent</span><span class="p">():</span> <span class="k">pass</span>
<span class="k">class</span> <span class="nc">Parent</span><span class="p">(</span><span class="n">GrandParent</span><span class="p">):</span> <span class="k">pass</span>
<span class="k">class</span> <span class="nc">Child</span><span class="p">(</span><span class="n">Parent</span><span class="p">):</span> <span class="k">pass</span>
</code></pre></div>
<p>calling the method <code>get_name()</code> on an instance of <code>Child</code> will look for it first into the <code>Child</code> class, then into <code>Parent</code> and <code>GrandParent</code>, in this order. Finally, it will check <code>object</code>.</p>
<p>What happens to the MRO when a class of this hierarchy defines a different metaclass? For example</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">NewType</span><span class="p">(</span><span class="nb">type</span><span class="p">):</span> <span class="k">pass</span>
<span class="k">class</span> <span class="nc">GrandParent</span><span class="p">():</span> <span class="k">pass</span>
<span class="k">class</span> <span class="nc">Parent</span><span class="p">(</span><span class="n">GrandParent</span><span class="p">):</span> <span class="k">pass</span>
<span class="k">class</span> <span class="nc">Child</span><span class="p">(</span><span class="n">Parent</span><span class="p">,</span> <span class="n">metaclass</span><span class="o">=</span><span class="n">NewType</span><span class="p">):</span> <span class="k">pass</span>
</code></pre></div>
<p>In this case everything works as usual, but after checking <code>object</code> the MRO will also check the <code>NewType</code> metaclass (and its ancestors).</p>
<p>So, metaclasses can act as mixins, and they are queried only at the end of the usual MRO. This is exactly what happens using multiple inheritance if <code>NewType</code> were a standard parent class that does not have <code>Parent</code> or <code>GrandParent</code> as ancestors.</p>
<p>Metaclasses are not part of the MRO however, since the MRO just deals with standard inheritance. If you check the MRO of the <code>Child</code> class, you will see that the metaclass is not included</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">Child</span><span class="o">.</span><span class="n">mro</span><span class="p">()</span>
<span class="go">[<class '__main__.Child'>, <class '__main__.Parent'>, <class '__main__.GrandParent'>, <class 'object'>]</span>
<span class="gp">>>> </span><span class="n">Child</span><span class="o">.</span><span class="vm">__class__</span>
<span class="go"><class '__main__.NewType'></span>
<span class="gp">>>> </span>
</code></pre></div>
<h2 id="abstract-methods">Abstract methods<a class="headerlink" href="#abstract-methods" title="Permanent link">¶</a></h2>
<p>Why are ABC called <code>abstract</code>? ABCs can be instantiated, so they are after all not pure interfaces (like Java ones, for example)</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">abc</span>
<span class="gp">>>> </span><span class="k">class</span> <span class="nc">MyABC</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">abc</span><span class="o">.</span><span class="n">ABCMeta</span><span class="p">):</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="n">m</span> <span class="o">=</span> <span class="n">MyABC</span><span class="p">()</span>
<span class="gp">>>> </span>
</code></pre></div>
<p>They may however define some methods as <em>abstract</em>, using the <code>abc.abstractmethod</code> decorator, which prevents the class from being instantiated if the method is not implemented. Let me give you a simple example: I define an Abstract Base Class with and abstract method</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">MyABC</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">abc</span><span class="o">.</span><span class="n">ABCMeta</span><span class="p">):</span>
<span class="nd">@abc</span><span class="o">.</span><span class="n">abstractmethod</span>
<span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">pass</span>
</code></pre></div>
<p>and try to instantiate it. Python complains</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">m</span> <span class="o">=</span> <span class="n">MyABC</span><span class="p">()</span>
<span class="gt">Traceback (most recent call last):</span>
File <span class="nb">"<stdin>"</span>, line <span class="m">1</span>, in <span class="n"><module></span>
<span class="gr">TypeError</span>: <span class="n">Can't instantiate abstract class MyABC with abstract methods get</span>
</code></pre></div>
<p>I am forced to create a new class that inherits from <code>MyABC</code> and implements the method</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">Concrete</span><span class="p">(</span><span class="n">MyABC</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="mi">1</span>
</code></pre></div>
<p>Now I can instantiate the class.</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">c</span> <span class="o">=</span> <span class="n">Concrete</span><span class="p">()</span>
<span class="gp">>>> </span>
</code></pre></div>
<p>Check the official documentation of Abstract Base Classes for a complete description of <code>@abstractmethod</code> and <code>@abstractproperty</code>.</p>
<h2 id="what-about-the-behaviour">What about the behaviour?<a class="headerlink" href="#what-about-the-behaviour" title="Permanent link">¶</a></h2>
<p>So, what happened to the Python tenet "Check the behaviour and not the structure"? With collections, after all, we dropped the EAFP protocol, going back to a Look Before You Leap approach. Are we going against the philosophy at the very base of the language?</p>
<p>It is very interesting to see what Guido van Rossum, creator of the Python language, says about this in <a href="http://legacy.python.org/dev/peps/pep-3119/">PEP 3119</a>: <em>Invocation means interacting with an object by invoking its methods. Usually this is combined with polymorphism, so that invoking a given method may run different code depending on the type of an object. Inspection means the ability for external code (outside of the object's methods) to examine the type or properties of that object, and make decisions on how to treat that object based on that information. [...] In classical OOP theory, invocation is the preferred usage pattern, and inspection is actively discouraged, being considered a relic of an earlier, procedural programming style. However, in practice this view is simply too dogmatic and inflexible, and leads to a kind of design rigidity that is very much at odds with the dynamic nature of a language like Python.</em></p>
<p>So the point is that forcing the use of a pure polymorphic approach sometimes can lead to solutions that are too complex or even incorrect. The key words here, in my opinion, are "dogmatic", "inflexible", and "rigidity", opposed to "dynamic nature". I really like this flexibility in a language and in its author.</p>
<p>Writing <code>if isinstance(obj, collections.Sequence)</code> is not EAFP, neither is any conditional test you may write. Nevertheless, no one would replace conditional tests with a pure EAFP approach, simply because sometimes those tests are more readable. This is the exact purpose of collections in Python and ABCs in general: to allow parts of the code to be simpler.</p>
<h2 id="final-words">Final words<a class="headerlink" href="#final-words" title="Permanent link">¶</a></h2>
<p>I hope this post helped you understand that Abstract Base Classes, and in particular the standard collections, are useful and easy to understand. Metaclasses are also not that scary and dangerous, even if using them obviously requires some skill.</p>
<h2 id="sources">Sources<a class="headerlink" href="#sources" title="Permanent link">¶</a></h2>
<p>The official documentation of the <code>abc</code> module is very well written. <a href="https://docs.python.org/3.5/library/abc.html">Here</a> you find the version for Python 3.5. I also suggest to read the original <a href="https://www.python.org/dev/peps/pep-3119">PEP 3119</a> and the related <a href="https://www.python.org/dev/peps/pep-3141">PEP 3141</a> for a deeper understanding of the topic.</p>
<h2 id="feedback">Feedback<a class="headerlink" href="#feedback" title="Permanent link">¶</a></h2>
<p>The <a href="https://github.com/TheDigitalCatOnline/blog_source/issues">GitHub issues</a> page is the best place to submit corrections.</p>Python decorators: metaprogramming with style2015-04-23T13:00:00+01:002015-04-23T13:00:00+01:00Leonardo Giordanitag:www.thedigitalcatonline.com,2015-04-23:/blog/2015/04/23/python-decorators-metaprogramming-with-style/<p>This post is the result of a lot of personal research on Python decorators, meta- and functional programming. I want however to thank Bruce Eckel and the people behind the open source book "Python 3 Patterns, Recipes and Idioms" for a lot of precious information on the subject. See the Resources section at the end of the post to check their work.</p>
<h2 id="is-python-functional">Is Python functional?<a class="headerlink" href="#is-python-functional" title="Permanent link">¶</a></h2>
<p>Well, no. Python is a strong object-oriented programming language and is not really going to mix OOP and functional like, for example, Scala (which is a very good language, by the way).</p>
<p>However, Python provides <em>some</em> features taken from functional programming. <a href="https://www.thedigitalcatonline.com/blog/2013/03/25/python-generators-from-iterators-to-cooperative-multitasking/">Generators and iterators</a> are one of them, and Python is not the only non pure functional programming language to have them in their toolbox.</p>
<p>Perhaps the most distinguishing feature of <a href="https://en.wikipedia.org/wiki/Functional_programming">functional languages</a> is that functions are first-class citizens (or first-class objects). This means that functions can be passed as an argument to other functions or can be returned by them. Functions, in functional languages, are just one of the data types available (even if this is a very rough simplification).</p>
<p>Python has three important features that allows it to provide a functional behaviour: references, function objects and callables.</p>
<h3 id="references">References<a class="headerlink" href="#references" title="Permanent link">¶</a></h3>
<p>Python variables share a common nature: they are all references. This means that variables are not typed per se, being pure memory addresses, and that functions do not declare the incoming data type for arguments (leaving aside gradual typing). Python polymorphism is based on delegation, and incoming function arguments are expected to provide a given behaviour, not a given structure.</p>
<p>Python functions are thus ready to accept every type of data that can be referenced, and functions can.</p>
<p>Read <a href="https://www.thedigitalcatonline.com/blog/2014/08/21/python-3-oop-part-4-polymorphism/">this post</a> to dive into delegation-based polymorphism and references in Python.</p>
<h2 id="functions-objects">Functions objects<a class="headerlink" href="#functions-objects" title="Permanent link">¶</a></h2>
<p>Since Python pushes the object-oriented paradigm to its maximum, it makes a point of always following the tenet <em>everything is an object</em>. So Python functions are objects as you can see from this simple example</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="k">def</span> <span class="nf">f</span><span class="p">():</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="nb">type</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="go"><class 'function'></span>
<span class="gp">>>> </span><span class="nb">type</span><span class="p">(</span><span class="nb">type</span><span class="p">(</span><span class="n">f</span><span class="p">))</span>
<span class="go"><class 'type'></span>
<span class="gp">>>> </span><span class="nb">type</span><span class="p">(</span><span class="n">f</span><span class="p">)</span><span class="o">.</span><span class="vm">__bases__</span>
<span class="go">(<class 'object'>,)</span>
<span class="gp">>>></span>
</code></pre></div>
<p>Given that, Python does nothing special to treat functions like first-class citizens, it simply recognizes that they are objects just like any other thing.</p>
<h3 id="callables">Callables<a class="headerlink" href="#callables" title="Permanent link">¶</a></h3>
<p>While Python has the well-defined <code>function</code> class seen in the above example, it relies more on the presence of the <code>__call__</code> method. That is, in Python any object can act as a function, provided that it has this method, which is invoked when the object is "called".</p>
<p>This will be crucial for the discussion about decorators, so be sure that you remember that we are usually more interested in <em>callable objects</em> and not only in functions, which, obviously, are a particular type of callable objects (or simply <em>callables</em>).</p>
<p>The fact that functions are callables can also be shown with some simple code</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="k">def</span> <span class="nf">f</span><span class="p">():</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="n">f</span><span class="o">.</span><span class="fm">__call__</span>
<span class="go"><method-wrapper '__call__' of function object at 0xb6709fa4></span>
</code></pre></div>
<h2 id="metaprogramming">Metaprogramming<a class="headerlink" href="#metaprogramming" title="Permanent link">¶</a></h2>
<p>While this is not a post on languages theory, it is worth spending a couple of words about metaprogramming. Usually "programming" can be seen as the task of applying transformations to data. Data and functions can be put together by an object-oriented approach, but they still are two different things. But you soon realize that, as you may run some code to change data, you may also run some code to change the code itself.</p>
<p>In low level languages this can be very simple, since at machine level everything is a sequence of bytes, and changing data or code does not make any difference. One of the most simple examples that I recall from my x86 Assembly years is the very simple self obfuscation code found is some computer viruses. The code was encrypted with a <a href="https://en.wikipedia.org/wiki/XOR_cipher">XOR cipher</a> and the first thing the code itself did upon execution was to decrypt its own code and then run it. The purpose of such tricks was (and is) to obfuscate the code such that it would be difficult for an antivirus to find the virus code and remove it. This is a very primitive form of metaprogramming, since it recognizes that for Assembly language there is no real distinction between code and data.</p>
<p>In higher lever languages such as Python achieving metaprogramming is no more a matter of changing byte values. It requires the language to treat its own structures as data. Every time we are trying to alter the behaviour of a language part we are actually metaprogramming. The first example that usually comes to mind are metaclasses (probably due to the "meta" word in their name), which are actually a way to change the default behaviour of the class creation process. Classes (part of the language) are created by another part of the language (metaclasses).</p>
<h2 id="decorators">Decorators<a class="headerlink" href="#decorators" title="Permanent link">¶</a></h2>
<p><a href="https://www.thedigitalcatonline.com/blog/2014/09/01/python-3-oop-part-5-metaclasses/">Metaclasses</a> are often perceived as a very tricky and dangerous thing to play with, and indeed they are seldom required in Python, with the most notable exception (no pun intended) being the <a href="https://www.thedigitalcatonline.com/blog/2014/09/04/python-3-oop-part-6-abstract-base-classes/">Abstract Base Classes</a> provided by the <code>collections</code> module.</p>
<p>Decorators, on the other side, are a feature loved by many experienced programmers and after their introduction the community has developed a big set of very interesting use cases.</p>
<p>I think that the first approach to decorators is often difficult for beginners because the functional version of decorators are indeed a bit complex to understand. Luckily, Python allows us to write decorators using classes too, which make the whole thing really easy to understand and write, I think.</p>
<p>So I will now review Python decorators starting from their rationale, then looking at class-based decorators without arguments, class-based decorators with arguments, and finally moving to function-based decorators.</p>
<h2 id="rationale">Rationale<a class="headerlink" href="#rationale" title="Permanent link">¶</a></h2>
<p>What are decorators, and why should you learn how to define and use them? </p>
<p>Well, decorators are a way to change the behaviour of a function or a class, so they are actually a way of metaprogramming, but they make it a lot more accessible than metaclasses do. Decorators are, in my opinion, a very <em>natural</em> way of altering functions and classes.</p>
<p>Moreover, with the addition of some syntactic sugar, they are a very compact way to both make changes and signal that those changes have been made.</p>
<p>The best syntactic form of a decorator is the following</p>
<div class="highlight"><pre><span></span><code><span class="nd">@dec</span>
<span class="k">def</span> <span class="nf">func</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwds</span><span class="p">):</span>
<span class="k">pass</span>
</code></pre></div>
<p>where <code>dec</code> is the name of the decorator and the function <code>func</code> is said to be <em>decorated</em> by it. As you can see any reader can quickly identify that the function has a special label attached, thus being altered in its behaviour.</p>
<p>This form, however, is just a simplification of the more generic form</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">func</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwds</span><span class="p">):</span>
<span class="k">pass</span>
<span class="n">func</span> <span class="o">=</span> <span class="n">dec</span><span class="p">(</span><span class="n">func</span><span class="p">)</span>
</code></pre></div>
<p>But what actually ARE the changes that you may want do to functions or classes? Let us stick for the moment to a very simple task: adding attributes. This is by no means a meaningless task, since there are many practical use cases that make use of it. Let us first test how we can add attributes to functions in plain Python</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="k">def</span> <span class="nf">func</span><span class="p">():</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="n">func</span><span class="o">.</span><span class="n">attr</span> <span class="o">=</span> <span class="s2">"a custom function attribute"</span>
<span class="gp">>>> </span><span class="n">func</span><span class="o">.</span><span class="n">attr</span>
<span class="go">'a custom function attribute'</span>
</code></pre></div>
<p>and to classes</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="k">class</span> <span class="nc">SomeClass</span><span class="p">:</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="n">SomeClass</span><span class="o">.</span><span class="n">attr</span> <span class="o">=</span> <span class="s2">"a custom class attribute"</span>
<span class="gp">>>> </span><span class="n">SomeClass</span><span class="o">.</span><span class="n">attr</span>
<span class="go">'a custom class attribute'</span>
<span class="gp">>>> </span><span class="n">s</span> <span class="o">=</span> <span class="n">SomeClass</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">s</span><span class="o">.</span><span class="n">attr</span>
<span class="go">'a custom class attribute'</span>
</code></pre></div>
<p>As you can see adding attributes to a class correctly results in a class attribute, which is thus shared by any instance of that class (check <a href="https://www.thedigitalcatonline.com/blog/2014/08/20/python-3-oop-part-2-classes-and-members/">this post</a> for some explanations about class attributes and sharing).</p>
<h2 id="class-based-decorators-without-arguments">Class-based decorators without arguments<a class="headerlink" href="#class-based-decorators-without-arguments" title="Permanent link">¶</a></h2>
<p>As already explained, Python allows you to call any object (as you do with functions) as long as it provides the <code>__call__()</code> method. So to write a class-based decorator you just need to create an object that defines such a method.</p>
<p>When used as a decorator, a class is instantiated at decoration time, that is when the function is defined, and called when the function is called.</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">CustomAttr</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">obj</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">attr</span> <span class="o">=</span> <span class="s2">"a custom function attribute"</span>
<span class="bp">self</span><span class="o">.</span><span class="n">obj</span> <span class="o">=</span> <span class="n">obj</span>
<span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">obj</span><span class="p">()</span>
</code></pre></div>
<p>As you can see there is already a lot of things that shall be clarified. First of all the class, when used as a decorator, is initialized with the object that is going to be decorated, here called <code>obj</code> (most of the time it is just called <code>f</code> for function, but you know that this is only a special case).</p>
<p>While the <code>__init__()</code> method is called at decoration time, the <code>__call__()</code> method of the decorator is called instead of the same method of the decorated object. In this case (decorator without arguments), the <code>__call__()</code> method of the decorator does not receive any argument. In this example we just "redirect" the call to the original function, which was stored during the initialization step.</p>
<p>So you see that in this case we have two different moments in which we may alter the behaviour of the decorated objects. The first is at its definition and the second is when it is actually called.</p>
<p>The decorator can be applied with the simple syntax shown in a previous section</p>
<div class="highlight"><pre><span></span><code><span class="nd">@CustomAttr</span>
<span class="k">def</span> <span class="nf">func</span><span class="p">():</span>
<span class="k">pass</span>
</code></pre></div>
<p>When Python parses the file and defines the function <code>func</code> the code it executes under the hood is</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">func</span><span class="p">():</span>
<span class="k">pass</span>
<span class="n">func</span> <span class="o">=</span> <span class="n">CustomAttr</span><span class="p">(</span><span class="n">func</span><span class="p">)</span>
</code></pre></div>
<p>according to the definition of decorator. This is why the class shall accept the decorated object as a parameter in its <code>__init__()</code> method.</p>
<p>Note that in this case the <code>func</code> object we obtain after the decoration is no more a function but a <code>CustomAttr</code> object</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">func</span>
<span class="go"><__main__.CustomAttr object at 0xb6f5ea8c></span>
</code></pre></div>
<p>and this is why in the <code>__init__()</code> method I attached the <code>attr</code> attribute to the class instance <code>self</code> and not to <code>obj</code>, so that now this works</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">func</span><span class="o">.</span><span class="n">attr</span>
<span class="go">'a custom function attribute'</span>
</code></pre></div>
<p>This replacement is also the reason why you shall also redefine <code>__call__()</code>. When you write <code>func()</code> you are not executing the function but calling the instance of <code>CustomAttr</code> returned by the decoration.</p>
<h2 id="class-based-decorators-with-arguments">Class-based decorators with arguments<a class="headerlink" href="#class-based-decorators-with-arguments" title="Permanent link">¶</a></h2>
<p>This case is the most natural step beyond the previous one. Once you started metaprogramming, you want to do it with style, and the first thing to do is to add parametrization. Adding parameters to decorators has the only purpose of generalizing the metaprogramming code, just like when you write parametrized functions instead of hardcoding values.</p>
<p>There is a big caveat here. Class-based decorators with arguments behave in a slightly different way to their counterpart without arguments. Specifically, the <code>__call__()</code> method is run during the decoration and not during the call.</p>
<p>Let us first review the syntax</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">CustomAttrArg</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">value</span> <span class="o">=</span> <span class="n">value</span>
<span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">obj</span><span class="p">):</span>
<span class="n">obj</span><span class="o">.</span><span class="n">attr</span> <span class="o">=</span> <span class="s2">"a custom function attribute with value </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">value</span><span class="p">)</span>
<span class="k">return</span> <span class="n">obj</span>
<span class="nd">@CustomAttrArg</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">func</span><span class="p">():</span>
<span class="k">pass</span>
</code></pre></div>
<p>Now the <code>__init__()</code> method shall accept some arguments, with the standard rules of Python functions for named and default arguments. The <code>__call__()</code> method receives the decorated object, which in the previous case was passed to the <code>__init__()</code> method.</p>
<p>The biggest change, however is that <code>__call__()</code> is not run when you call the decorated object, but immediately after <code>__init__()</code> during the decoration phase. This results in the following difference: while in the previous case the decorated object was no more itself, but an instance of the decorator, now the decorated objects becomes the return value of the <code>__call__()</code> method.</p>
<p>Remember that, when you call the decorated object, you are now actually calling what you get from <code>__call__()</code> so be sure of returning something meaningful.</p>
<p>In the above example I stored one single argument in <code>__init__()</code>, and this argument is passed to the decorator when applying it to the function. Then, in <code>__call__()</code>, I set the attribute of the decorated object, using the stored argument, and return the object itself. This is important, since I have to return a callable object.</p>
<p>This means that, if you have to do something complex with the decorated object, you may just define a local function that makes use of it and return this function. Let us see a very simple example</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">CustomAttrArg</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">value</span> <span class="o">=</span> <span class="n">value</span>
<span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">obj</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">wrap</span><span class="p">():</span>
<span class="c1"># Here you can do complex stuff</span>
<span class="n">obj</span><span class="p">()</span>
<span class="c1"># Here you can do complex stuff</span>
<span class="k">return</span> <span class="n">wrap</span>
<span class="nd">@CustomAttrArg</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">func</span><span class="p">():</span>
<span class="k">pass</span>
</code></pre></div>
<p>Here the returned object is no more the decorated one, but a new function <code>wrap()</code> defined locally. It is interesting to show how Python identifies it</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="nd">@CustomAttrArg</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="gp">... </span><span class="k">def</span> <span class="nf">func</span><span class="p">():</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="n">func</span>
<span class="go"><function CustomAttrArg.__call__.<locals>.wrap at 0xb70185cc></span>
</code></pre></div>
<p>This pattern enables you to do every sort of things with the decorated object. Not only to change it (adding attributes, for example), but also pre- or post- filtering its results. You may start to understand that what we called metaprogramming may be very useful for everyday tasks, and not only for some obscure wizardry.</p>
<h3 id="decorators-and-prototypes">Decorators and prototypes<a class="headerlink" href="#decorators-and-prototypes" title="Permanent link">¶</a></h3>
<p>If you write a class-based decorator with arguments, you are in charge of returning a callable object of choice. There is no assumption on the returned object, even if the usual case is that the returned object has the same prototype as the decorated one.</p>
<p>This means that if the decorated object accepts zero arguments (like in my example), you usually return a callable that accepts zero arguments. This is however by no means enforced by the language, and through this you may push the metaprogramming technique a bit. I'm not providing examples of this technique in this post, however.</p>
<h2 id="function-based-decorators">Function-based decorators<a class="headerlink" href="#function-based-decorators" title="Permanent link">¶</a></h2>
<p>Function-based decorators are very simple for simple cases and a bit trickier for complex ones. The problem is that their syntax can be difficult to grasp at first sight if you never saw a decorator. They are indeed not very different from the class-based decorators with arguments case, as they define a local function that wraps the decorated object and return it.</p>
<p>The case without arguments is always the simplest one</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">decorate</span><span class="p">(</span><span class="n">f</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">wrap</span><span class="p">():</span>
<span class="n">f</span><span class="p">()</span>
<span class="k">return</span> <span class="n">wrap</span>
<span class="nd">@decorate</span>
<span class="k">def</span> <span class="nf">func</span><span class="p">():</span>
<span class="k">pass</span>
</code></pre></div>
<p>This behaves like the equivalent case with classes. The function is passed as an argument to the <code>decorate()</code> function by the decoration process that calls it passing the decorated object. When you actually call the function, however, you are actually calling <code>wrap()</code>.</p>
<p>As happens for class-based decorators, the parametrization changes the calling procedure. This is the code</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">decorate</span><span class="p">(</span><span class="n">arg1</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">wrap</span><span class="p">(</span><span class="n">f</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">_wrap</span><span class="p">(</span><span class="n">arg</span><span class="p">):</span>
<span class="n">f</span><span class="p">(</span><span class="n">arg</span> <span class="o">+</span> <span class="n">arg1</span><span class="p">)</span>
<span class="k">return</span> <span class="n">_wrap</span>
<span class="k">return</span> <span class="n">wrap</span>
<span class="nd">@decorate</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">func</span><span class="p">(</span><span class="n">arg</span><span class="p">):</span>
<span class="k">pass</span>
</code></pre></div>
<p>As you see it is not really straightforward, and this is the reason I preferred to discuss it as the last case. Recall what we learned about class-based decorators: the first call to the <code>decorate()</code> function happens when the decorator is called with an argument. Thus <code>@decorate(1)</code> calls <code>decorate()</code> passing <code>1</code> as <code>arg1</code>, and this function returns the <code>wrap()</code> local function.</p>
<p>This second function accepts another function as an argument, and indeed it is used in the actual decoration process, which can be represented by the code <code>func = wrap(func)</code>. This <code>wrap()</code> function, being used to decorate <code>func()</code>, wants to return a compatible object, that is in this case a function that accepts a single argument. This is why, in turn, <code>wrap()</code> defines and returns a <code>_wrap()</code> local function, which eventually uses both the argument passed to <code>func()</code> and the argument passed to the decorator.</p>
<p>So the process may be summarized as follows (I will improperly call <code>func_dec</code> the decorated function to show what is happening)</p>
<ul>
<li><code>@decorator(1)</code> returns a <code>wrap()</code> function (that knows the argument)</li>
<li><code>func</code> is redefined as <code>func_dec = wrap(func)</code> becoming <code>_wrap()</code></li>
<li>When you call <code>func_dec(arg)</code> Python executes <code>_wrap(arg)</code> which calls the original <code>func()</code> passing <code>1 + arg</code> as argument</li>
</ul>
<p>Obviously the power of the decorator concept is that you are not dealing with <code>func_dec()</code> but with <code>func()</code> itself, and all the "magic" happens under the hood.</p>
<p>If you feel uncomfortable with function-based decorators don't worry, as they are indeed a bit awkward. I usually stick to function based decorators for the simple cases (setting class attributes, for example), and move to class-based ones when the code is more complex.</p>
<h2 id="example">Example<a class="headerlink" href="#example" title="Permanent link">¶</a></h2>
<p>A good example of the power of decorators which comes out of the box is <code>functools.total_ordering</code>. The <code>functools</code> module provides a lot of interesting tools to push the functional approach in Python, most notably <code>partial()</code> and <code>partialmethod()</code>, which are however out of the scope of this post.</p>
<p>The <code>total_ordering</code> decorator (documented <a href="https://docs.python.org/3.4/library/functools.html#functools.total_ordering">here</a>) wants to make an object provide a full set of comparison ordering methods starting from a small set of them. Comparison methods are those methods Python calls when two objects are compared. For example when you write <code>a == b</code>, Python executes <code>a.__eq__(b)</code> and the same happens for the other five operators <code>></code> (<code>__gt__</code>), <code><</code> (<code>__lt__</code>), <code>>=</code> (<code>__ge__</code>), <code><=</code> (<code>__le__</code>) and <code>!=</code> (<code>__ne__</code>).</p>
<p>Mathematically all those operators may be expressed using only one of them and the <code>__eq__()</code> method, for example <code>__ne__</code> is <code>!__eq__</code> and <code>__lt__</code> is <code>__le__ and !__eq__</code>. This decorator makes use of this fact to provide the missing methods for the decorated object. A quick example</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">Person</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
<span class="k">def</span> <span class="fm">__eq__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">==</span> <span class="n">other</span><span class="o">.</span><span class="n">name</span>
<span class="k">def</span> <span class="fm">__lt__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o"><</span> <span class="n">other</span><span class="o">.</span><span class="n">name</span>
</code></pre></div>
<p>This is a simple class that defines the <code>==</code> and <code><</code> comparison methods.</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">p1</span> <span class="o">=</span> <span class="n">Person</span><span class="p">(</span><span class="s1">'Bob'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">p2</span> <span class="o">=</span> <span class="n">Person</span><span class="p">(</span><span class="s1">'Alice'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">p1</span> <span class="o">==</span> <span class="n">p2</span>
<span class="go">False</span>
<span class="gp">>>> </span><span class="n">p1</span> <span class="o"><</span> <span class="n">p2</span>
<span class="go">False</span>
<span class="gp">>>> </span><span class="n">p1</span> <span class="o">>=</span> <span class="n">p2</span>
<span class="gt">Traceback (most recent call last):</span>
File <span class="nb">"/home/leo/prova.py"</span>, line <span class="m">103</span>, in <span class="n"><module></span>
<span class="w"> </span><span class="n">p1</span> <span class="o">>=</span> <span class="n">p2</span>
<span class="gr">TypeError</span>: <span class="n">unorderable types: Person() >= Person()</span>
</code></pre></div>
<p>A big warning: Python doesn't complain if you try to perform the <code>></code> and <code>!=</code> comparisons but lacking the dedicated methods it does perform a "standard" comparison. This means that, as the documentation states <a href="https://docs.python.org/2/reference/datamodel.html#object.__lt__">here</a>, "There are no implied relationships among the comparison operators. The truth of x==y does not imply that x!=y is false."</p>
<p>With the <code>total_ordering</code> decorator, however, all six comparisons become available</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">functools</span>
<span class="nd">@functools</span><span class="o">.</span><span class="n">total_ordering</span>
<span class="k">class</span> <span class="nc">Person</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
<span class="k">def</span> <span class="fm">__eq__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">==</span> <span class="n">other</span><span class="o">.</span><span class="n">name</span>
<span class="k">def</span> <span class="fm">__lt__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o"><</span> <span class="n">other</span><span class="o">.</span><span class="n">name</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">p1</span> <span class="o">=</span> <span class="n">Person</span><span class="p">(</span><span class="s1">'Bob'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">p2</span> <span class="o">=</span> <span class="n">Person</span><span class="p">(</span><span class="s1">'Alice'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">p1</span> <span class="o">==</span> <span class="n">p2</span>
<span class="go">False</span>
<span class="gp">>>> </span><span class="n">p1</span> <span class="o">!=</span> <span class="n">p2</span>
<span class="go">True</span>
<span class="gp">>>> </span><span class="n">p1</span> <span class="o">></span> <span class="n">p2</span>
<span class="go">True</span>
<span class="gp">>>> </span><span class="n">p1</span> <span class="o"><</span> <span class="n">p2</span>
<span class="go">False</span>
<span class="gp">>>> </span><span class="n">p1</span> <span class="o">>=</span> <span class="n">p2</span>
<span class="go">True</span>
<span class="gp">>>> </span><span class="n">p1</span> <span class="o"><=</span> <span class="n">p2</span>
<span class="go">False</span>
</code></pre></div>
<h2 id="final-words">Final words<a class="headerlink" href="#final-words" title="Permanent link">¶</a></h2>
<p>Decorators are a very powerful tool, and they are worth learning. This may be for you just the first step into the amazing world of metaprogramming or just an advanced technique you may use to simplify your code. Whatever you do with them be sure to understand the difference between the two cases (with and without arguments) and don't avoid function-based decorators just because their syntax is a bit complex.</p>
<h2 id="resources">Resources<a class="headerlink" href="#resources" title="Permanent link">¶</a></h2>
<p>Many thanks to Bruce Eckel for <a href="http://www.artima.com/weblogs/viewpost.jsp?thread=240808">his three posts</a> which have been (I think) the source for the page on <a href="http://python-3-patterns-idioms-test.readthedocs.org/en/latest/PythonDecorators.html">Python 3 Patterns, Recipes and Idioms</a> (the latter is still work in progress). Update: as Bruce stated "the Python3 Patterns book is kind of a failed project". So beware that information contained there is not going to be updated. It is however still a good starting point for Python studies and investigations.</p>
<p>A good source of advanced decorators can be found at the <a href="https://wiki.python.org/moin/PythonDecoratorLibrary">Python Decorator Library</a>, and a lot of stuff may be found on Stackoverflow under the <a href="https://stackoverflow.com/questions/tagged/python-decorators">python-decorators tag</a>.</p>
<p><a href="https://github.com/GrahamDumpleton">Graham Dumpleton</a> wrote a very interesting a in-depth analysis of Python decorators <a href="https://github.com/GrahamDumpleton/wrapt/tree/develop/blog">here</a>.</p>
<h2 id="feedback">Feedback<a class="headerlink" href="#feedback" title="Permanent link">¶</a></h2>
<p>The <a href="https://github.com/TheDigitalCatOnline/blog_source/issues">GitHub issues</a> page is the best place to submit corrections.</p>Advanced use of Python decorators and metaclasses2014-10-14T09:43:08+02:002014-10-14T09:43:08+02:00Leonardo Giordanitag:www.thedigitalcatonline.com,2014-10-14:/blog/2014/10/14/decorators-and-metaclasses/<h2 id="abstract">Abstract<a class="headerlink" href="#abstract" title="Permanent link">¶</a></h2>
<p>While introducing people to Python metaclasses I realized that sometimes the big problem of the most powerful Python features is that programmers do not perceive how they may simplify their usual tasks. Therefore, features like metaclasses are considered a fancy but rather unuseful addition to a standard OOP language, instead of a real game changer.</p>
<p>This post wants to show how to use metaclasses and decorators to create a powerful class that can be inherited and customized by easily adding decorated methods.</p>
<h2 id="metaclasses-and-decorators-a-match-made-in-space">Metaclasses and decorators: a match made in space<a class="headerlink" href="#metaclasses-and-decorators-a-match-made-in-space" title="Permanent link">¶</a></h2>
<p>Metaclasses are a complex topic, and most of the times even advanced programmers do not see a wide range of practical uses for them. Chances are that this is the part of Python (or other languages that support metaclasses, like Smalltalk and Ruby) that fits the least the "standard" object-oriented patterns or solutions found in C++ and Java, just to mention two big players.</p>
<p>Indeed metaclasess usually come in play when programming advanced libraries or frameworks, where a lot of automation must be provided. For example, Django Forms system heavily relies on metaclasses to provide all its magic.</p>
<p>We also have to note, however, that we usually call "magic" or "tricks" all those techniques we are not familiar with, and as a result in Python many things are called this way, being its implementation often peculiar compared to other languages.</p>
<p>Time to bring some spice into your programming: let's practice some Python wizardry and exploit the power of the language!</p>
<p>In this post I want to show you an interesting joint use of decorators and metaclasses. I will show you how to use decorators to mark methods so that they can be automatically used by the class when performing a given operation.</p>
<p>More in detail, I will implement a class that can be called on a string to "process" it, and show you how to implement different "filters" through simple decorated methods. What I want to obtain is something like this:</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">MyStringProcessor</span><span class="p">(</span><span class="n">StringProcessor</span><span class="p">):</span>
<span class="nd">@stringfilter</span>
<span class="k">def</span> <span class="nf">capitalize</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="nd">@stringfilter</span>
<span class="k">def</span> <span class="nf">remove_double_spaces</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="n">msp</span> <span class="o">=</span> <span class="n">MyStringProcessor</span><span class="p">()</span>
<span class="s2">"A test string"</span> <span class="o">==</span> <span class="n">msp</span><span class="p">(</span><span class="s2">"a test string"</span><span class="p">)</span>
</code></pre></div>
<p>The module defines a <code>StringProcessor</code> class that I can inherit and customize adding methods that have a standard signature <code>(self, str)</code> and are decorated with <code>@stringfilter</code>. This class can later be instantiated and the instance used to directly process a string and return the result. Internally the class automatically executes all the decorated methods in succession. I also would like the class to obey the order I defined the filters: first defined, first executed.</p>
<h2 id="the-hitchhikers-guide-to-metaclasses">The Hitchhiker's Guide To Metaclasses<a class="headerlink" href="#the-hitchhikers-guide-to-metaclasses" title="Permanent link">¶</a></h2>
<p>How can metaclasses help to reach this target?</p>
<p>Simply put, metaclasses are classes that are instantiated to get classes. That means that whenever I use a class, for example to instantiate it, first Python <em>builds</em> that class using the metaclass and the class definition we wrote. For example, you know that you can find the class members in the <code>__dict__</code> attribute: this attribute is created by the standard metaclass, which is <code>type</code>.</p>
<p>Given that, a metaclass is a good starting point for us to insert some code to identify a subset of functions inside the definition of the class. In other words, we want the output of the metaclass (that is, the class) be built exactly as happens in the standard case, but with an addition: a separate list of all the methods decorated with <code>@stringfilter</code>.</p>
<p>You know that a class has a <em>namespace</em>, that is a dictionary of what was defined inside the class. So, when the standard <code>type</code> metaclass is used to create a class, the class body is parsed and a <code>dict()</code> object is used to collect the namespace.</p>
<p>We are however interested in preserving the order of definition and a Python dictionary is an unordered structure, so we take advantage of the <code>__prepare__</code> hook introduced in the class creation process with Python 3. This function, if present in the metaclass, is used to preprocess the class and to return the structure used to host the namespace. So, following the example found in the official documentation, we start defining a metaclass like</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">FilterClass</span><span class="p">(</span><span class="nb">type</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__prepare__</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">bases</span><span class="p">,</span> <span class="o">**</span><span class="n">kwds</span><span class="p">):</span>
<span class="k">return</span> <span class="n">collections</span><span class="o">.</span><span class="n">OrderedDict</span><span class="p">()</span>
</code></pre></div>
<p>This way, when the class will be created, an <code>OrderedDict</code> will be used to host the namespace, allowing us to keep the definition order. Please note that the signature <code>__prepare__(name, bases, **kwds)</code> is enforced by the language. If you want the method to get the metaclass as a first argument (because the code of the method needs it) you have to change the signature to <code>__prepare__(metacls, name, bases, **kwds)</code> and decorate it with <code>@classmethod</code>.</p>
<p>The second function we want to define in our metaclass is <code>__new__</code>. Just like happens for the instantiation of classes, this method is invoked by Python to get a new instance of the metaclass, and is run before <code>__init__</code>. Its signature has to be <code>__new__(metacls, name, bases, namespace, **kwds)</code> and the result shall be an instance of the metaclass. As for its normal class counterpart (after all a metaclass is a class), <code>__new__()</code> usually wraps the same method of the parent class, <code>type</code> in this case, adding its own customizations.</p>
<p>The customization we need is the creation of a list of methods that are marked in some way (the decorated filters). Say for simplicity's sake that the decorated methods have an attribute <code>_filter</code>.</p>
<p>The full metaclass is then</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">FilterClass</span><span class="p">(</span><span class="nb">type</span><span class="p">):</span>
<span class="nd">@classmethod</span>
<span class="k">def</span> <span class="fm">__prepare__</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">bases</span><span class="p">,</span> <span class="o">**</span><span class="n">kwds</span><span class="p">):</span>
<span class="k">return</span> <span class="n">collections</span><span class="o">.</span><span class="n">OrderedDict</span><span class="p">()</span>
<span class="k">def</span> <span class="fm">__new__</span><span class="p">(</span><span class="n">metacls</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">bases</span><span class="p">,</span> <span class="n">namespace</span><span class="p">,</span> <span class="o">**</span><span class="n">kwds</span><span class="p">):</span>
<span class="n">result</span> <span class="o">=</span> <span class="nb">type</span><span class="o">.</span><span class="fm">__new__</span><span class="p">(</span><span class="n">metacls</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">bases</span><span class="p">,</span> <span class="nb">dict</span><span class="p">(</span><span class="n">namespace</span><span class="p">))</span>
<span class="n">result</span><span class="o">.</span><span class="n">_filters</span> <span class="o">=</span> <span class="p">[</span>
<span class="n">value</span> <span class="k">for</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">namespace</span><span class="o">.</span><span class="n">values</span><span class="p">()</span> <span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="s1">'_filter'</span><span class="p">)]</span>
<span class="k">return</span> <span class="n">result</span>
</code></pre></div>
<p>Now we have to find a way to mark all filter methods with a <code>_filter</code> attribute.</p>
<h2 id="the-anatomy-of-purple-decorators">The Anatomy of Purple Decorators<a class="headerlink" href="#the-anatomy-of-purple-decorators" title="Permanent link">¶</a></h2>
<p><strong>decorate</strong>: <em>to add something to an object or place, especially in order to make it more attractive</em> (Cambridge Dictionary)</p>
<p>Decorators are, as the name suggests, the best way to augment functions or methods. Remember that a decorator is basically a callable that accepts another callable, processes it, and returns it.</p>
<p>Used in conjunction with metaclasses, decorators are a very powerful and expressive way to implement advanced behaviours in our code. In this case we may easily use them to add an attribute to decorated methods, one of the most basic tasks for a decorator.</p>
<p>I decided to implement the <code>@stringfilter</code> decorator as a function, even if I usually prefer implementing them as classes. The reason is that decorator classes behave differently when used to implement decorators without arguments rather than decorators with arguments. In this case this difference would force to write some complex code and an explanation of that would be overkill now. In a future post on dectorators you will find all the gory details, but in the meantime you may check the three Bruce Eckel posts listed in the references section.</p>
<p>The decorator is very simple:</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">stringfilter</span><span class="p">(</span><span class="n">func</span><span class="p">):</span>
<span class="n">func</span><span class="o">.</span><span class="n">_filter</span> <span class="o">=</span> <span class="kc">True</span>
<span class="k">return</span> <span class="n">func</span>
</code></pre></div>
<p>As you can see the decorator just creates an attribute called <code>_filter</code> into the function (remember that functions are objects). The actual value of this attribute is not important in this case, since we are just interested in telling apart class members that contain it.</p>
<h2 id="the-dynamics-of-a-callable-object">The Dynamics of a Callable Object<a class="headerlink" href="#the-dynamics-of-a-callable-object" title="Permanent link">¶</a></h2>
<p>We are used to think about functions as special language components that may be "called" or executed. In Python functions are objects, just like everything else, and the feature that allows them to be executed comes from the presence of the <code>__call__()</code> method. Python is polymorphic by design and based on delegation, so (almost) everything that happens in the code relies on some features of the target object.</p>
<p>The result of this generalization is that every object that contains the <code>__call__()</code> method may be executed like a function, and gains the name of <em>callable object</em>.</p>
<p>The <code>StringProcessor</code> class shall thus contain this method and perform there the string processing with all the contained filters. The code is</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">StringProcessor</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">FilterClass</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">string</span><span class="p">):</span>
<span class="n">_string</span> <span class="o">=</span> <span class="n">string</span>
<span class="k">for</span> <span class="n">_filter</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_filters</span><span class="p">:</span>
<span class="n">_string</span> <span class="o">=</span> <span class="n">_filter</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">_string</span><span class="p">)</span>
<span class="k">return</span> <span class="n">_string</span>
</code></pre></div>
<p>A quick review of this simple function shows that it accepts the string as an argument, stores it in a local variable and loops over the filters, executing each of them on the local string, that is on the result of the previous filter.</p>
<p>The filter functions are extracted from the <code>self._filters</code> list, that is compiled by the <code>FilterClass</code> metaclass we already discussed.</p>
<p>What we need to do now is to inherit from <code>StringProcessor</code> to get the metaclass machinery and the <code>__call__()</code> method, and to define as many methods as needed, decorating them with the <code>@stringfilter</code> decorator.</p>
<p>Note that, thanks to the decorator and the metaclass, you may have other methods in your class that do not interfere with the string processing as long as they are not decorated with the decorator under consideration.</p>
<p>An example derived class may be the following</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">MyStringProcessor</span><span class="p">(</span><span class="n">StringProcessor</span><span class="p">):</span>
<span class="nd">@stringfilter</span>
<span class="k">def</span> <span class="nf">capitalize</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">string</span><span class="p">):</span>
<span class="k">return</span> <span class="n">string</span><span class="o">.</span><span class="n">capitalize</span><span class="p">()</span>
<span class="nd">@stringfilter</span>
<span class="k">def</span> <span class="nf">remove_double_spaces</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">string</span><span class="p">):</span>
<span class="k">return</span> <span class="n">string</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">' '</span><span class="p">,</span> <span class="s1">' '</span><span class="p">)</span>
</code></pre></div>
<p>The two <code>capitalize()</code> and <code>remove_double_spaces()</code> methods have been decorated, so they will be applied in order to any string passed when calling the class. A quick example of this last class is</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">strproc</span>
<span class="gp">>>> </span><span class="n">msp</span> <span class="o">=</span> <span class="n">strproc</span><span class="o">.</span><span class="n">MyStringProcessor</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">input_string</span> <span class="o">=</span> <span class="s2">"a test string"</span>
<span class="gp">>>> </span><span class="n">output_string</span> <span class="o">=</span> <span class="n">msp</span><span class="p">(</span><span class="n">input_string</span><span class="p">)</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s2">"INPUT STRING:"</span><span class="p">,</span> <span class="n">input_string</span><span class="p">)</span>
<span class="go">INPUT STRING: a test string</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s2">"OUTPUT STRING:"</span><span class="p">,</span> <span class="n">output_string</span><span class="p">)</span>
<span class="go">OUTPUT STRING: A test string</span>
<span class="gp">>>> </span>
</code></pre></div>
<p>That's it!</p>
<h2 id="final-words">Final words<a class="headerlink" href="#final-words" title="Permanent link">¶</a></h2>
<p>There are obviously other ways to accomplish the same task, and this post wanted just to give a practical example of what metaclasses are good for, and why I think that they should be part of any Python programmer's arsenal.</p>
<p>[Update] Some developers <a href="https://www.reddit.com/r/Python/comments/2jbi2f/advanced_use_of_python_decorators_and_metaclasses/">on Reddit</a> and Linkedin raised objections to the content of the post mainly about the fact that the example may be perfectly implemented without metaclasses and about the dangerous nature of metaclasses. Since I try to learn from everyone, I thank them for their suggestions.</p>
<p>It is especially interesting to know that some developers consider the use of metaclasses a risky business, because they hide a lot of the structure of the class and the underlying machinery. This is true, so (as you should do for other technologies), think carefully about the reasons that drive you to use metaclasses, and be sure you know them well.</p>
<h2 id="book-trivia">Book Trivia<a class="headerlink" href="#book-trivia" title="Permanent link">¶</a></h2>
<p>Section titles come from the following books: <em>A Match Made in Space - George McFly</em>, <em>The Hitchhiker's Guide To the Galaxy - Various Authors</em>, <em>The Anatomy of Purple Dragons - Unknown</em>, <em>The Dynamics of an Asteroid - James Moriarty</em>.</p>
<h2 id="source-code">Source code<a class="headerlink" href="#source-code" title="Permanent link">¶</a></h2>
<p>The <a href="/code/metaclasses/strproc.py">strproc.py</a> file contains the full source code used in this post.</p>
<h2 id="online-resources">Online resources<a class="headerlink" href="#online-resources" title="Permanent link">¶</a></h2>
<p>The following resources may be useful.</p>
<h4 id="metaclasses">Metaclasses<a class="headerlink" href="#metaclasses" title="Permanent link">¶</a></h4>
<ul>
<li>Python 3 official documentation: <a href="https://docs.python.org/3.4/reference/datamodel.html#customizing-class-creation">customizing class creation</a>.</li>
<li>Python 3 OOP Part 5 - Metaclasses <a href="https://www.thedigitalcatonline.com/blog/2014/09/01/python-3-oop-part-5-metaclasses/">on this blog</a>.</li>
<li><a href="http://python-3-patterns-idioms-test.readthedocs.org/en/latest/Metaprogramming.html">Metaprogramming examples and patterns</a> (still using some Python 2 code but useful).</li>
</ul>
<h4 id="decorators">Decorators<a class="headerlink" href="#decorators" title="Permanent link">¶</a></h4>
<ul>
<li><a href="http://www.artima.com/weblogs/viewpost.jsp?thread=240808">Bruce Eckel</a> on decorators (series of three posts, 6 years old but still valid).</li>
<li><a href="http://simeonfranklin.com/blog/2012/jul/1/python-decorators-in-12-steps/">A different approach</a> on explaining decorators.</li>
<li><a href="http://www.jeffknupp.com/blog/2013/11/29/improve-your-python-decorators-explained/">Jeff Knupp</a> goes deep inside the concept of function.</li>
</ul>
<h4 id="callable-objects">Callable objects<a class="headerlink" href="#callable-objects" title="Permanent link">¶</a></h4>
<ul>
<li><a href="http://www.rafekettler.com/magicmethods.html#callable">Rafe Kettler</a> provides a very detaild guide on Python "magic" methods.</li>
</ul>
<h2 id="updates">Updates<a class="headerlink" href="#updates" title="Permanent link">¶</a></h2>
<p>2014-10-17: <a href="https://github.com/thermokarst">Matthew Dillon</a> and <a href="https://github.com/dmcdougall">Damon McDougall</a> spotted two typos. Thank you!</p>
<p>2014-10-17: <a href="https://www.reddit.com/user/ionelmc">ionelmc</a> suggested two corrections <a href="https://www.reddit.com/r/Python/comments/2jbi2f/advanced_use_of_python_decorators_and_metaclasses/cla696y">here</a> and <a href="https://www.reddit.com/r/Python/comments/2jbi2f/advanced_use_of_python_decorators_and_metaclasses/cla6o77">here</a>. Both are correct so I implemented them. The second one is more about style, but fits well the introductory purpose of the post. Thanks!</p>
<h2 id="feedback">Feedback<a class="headerlink" href="#feedback" title="Permanent link">¶</a></h2>
<p>The <a href="https://github.com/TheDigitalCatOnline/blog_source/issues">GitHub issues</a> page is the best place to submit corrections.</p>Object-Oriented Programming in Python 3 - Abstract Base Classes2014-09-04T10:00:00+02:002019-05-22T23:30:00+00:00Leonardo Giordanitag:www.thedigitalcatonline.com,2014-09-04:/blog/2014/09/04/python-3-oop-part-6-abstract-base-classes/<p>Fundamentals of object-oriented programming in Python - abstract base classes</p><p>This post is available as an <strong>IPython Notebook</strong> <a href="/notebooks/Python_3_OOP_Part_6__Abstract_Base_Classes.ipynb">here</a></p>
<h2 id="the-inspection-club">The Inspection Club<a class="headerlink" href="#the-inspection-club" title="Permanent link">¶</a></h2>
<p>As you know, Python leverages polymorphism at its maximum by dealing only with generic references to objects. This makes OOP not an addition to the language but part of its structure from the ground up. Moreover, Python pushes the EAFP appoach, which tries to avoid direct inspection of objects as much as possible.</p>
<p>It is however very interesting to read what Guido van Rossum says in <a href="http://legacy.python.org/dev/peps/pep-3119/">PEP 3119</a>: <em>Invocation means interacting with an object by invoking its methods. Usually this is combined with polymorphism, so that invoking a given method may run different code depending on the type of an object. Inspection means the ability for external code (outside of the object's methods) to examine the type or properties of that object, and make decisions on how to treat that object based on that information. [...] In classical OOP theory, invocation is the preferred usage pattern, and inspection is actively discouraged, being considered a relic of an earlier, procedural programming style. However, in practice this view is simply too dogmatic and inflexible, and leads to a kind of design rigidity that is very much at odds with the dynamic nature of a language like Python.</em></p>
<p>The author of Python recognizes that forcing the use of a pure polymorphic approach leads sometimes to solutions that are too complex or even incorrect. In this section I want to show some of the problems that can arise from a pure polymorphic approach and introduce Abstract Base Classes, which aim to solve them. I strongly suggest to read <a href="http://legacy.python.org/dev/peps/pep-3119/">PEP 3119</a> (as for any other PEP) since it contains a deeper and better explanation of the whole matter. Indeed I think that this PEP is so well written that any further explanation is hardly needed. I am however used to write explanations to check how much I understood about the topic, so I am going to try it this time too.</p>
<h4 id="eafp-the-extra-test-trial">E.A.F.P the Extra Test Trial<a class="headerlink" href="#eafp-the-extra-test-trial" title="Permanent link">¶</a></h4>
<p>The EAFP coding style requires you to trust the incoming objects to provide the attributes and methods you need, and to manage the possible exceptions, if you know how to do it. Sometimes, however, you need to test if the incoming object matches a complex behaviour. For example, you could be interested in testing if the object <em>acts</em> like a list, but you quickly realize that the amount of methods a <code>list</code> provides is very big and this could lead to odd EAFP code like</p>
<div class="highlight"><pre><span></span><code><span class="k">try</span><span class="p">:</span>
<span class="n">obj</span><span class="o">.</span><span class="n">append</span>
<span class="n">obj</span><span class="o">.</span><span class="n">count</span>
<span class="n">obj</span><span class="o">.</span><span class="n">extend</span>
<span class="n">obj</span><span class="o">.</span><span class="n">index</span>
<span class="n">obj</span><span class="o">.</span><span class="n">insert</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="k">except</span> <span class="ne">AttributeError</span><span class="p">:</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
</code></pre></div>
<p>where the methods of the <code>list</code> type are accessed (not called) just to force the object to raise the <code>AttributeError</code> exception if they are not present. This code, however, is not only ugly but also wrong. If you recall the "Enter the Composition" section of the <a href="https://www.thedigitalcatonline.com/blog/2014/08/20/python-3-oop-part-3-delegation-composition-and-inheritance/">third post</a> of this series, you know that in Python you can always customize the <code>__getattr__()</code> method, which is called whenever the requested attribute is not found in the object. So I could write a class that passes the test but actually does not act like a list</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">FakeList</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">fakemethod</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">pass</span>
<span class="k">def</span> <span class="fm">__getattr__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="k">if</span> <span class="n">name</span> <span class="ow">in</span> <span class="p">[</span><span class="s1">'append'</span><span class="p">,</span> <span class="s1">'count'</span><span class="p">,</span> <span class="s1">'extend'</span><span class="p">,</span> <span class="s1">'index'</span><span class="p">,</span> <span class="s1">'insert'</span><span class="p">,</span> <span class="o">...</span><span class="p">]:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">fakemethod</span>
</code></pre></div>
<p>This is obviously just an example, and no one will ever write such a class, but this demonstrates that just accessing methods does not guarantee that a class <em>acts</em> like the one we are expecting.</p>
<p>There are many examples that could be done leveraging the highly dynamic nature of Python and its rich object model. I would summarize them by saying that sometimes you'd better to check the type of the incoming object.</p>
<p>In Python you can obtain the type of an object using the <code>type()</code> built-in function, but to check it you'd better use <code>isinstance()</code>, which returns a boolean value. Let us see an example before moving on</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="nb">isinstance</span><span class="p">([],</span> <span class="nb">list</span><span class="p">)</span>
<span class="go">True</span>
<span class="gp">>>> </span><span class="nb">isinstance</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="nb">int</span><span class="p">)</span>
<span class="go">True</span>
<span class="gp">>>> </span><span class="k">class</span> <span class="nc">Door</span><span class="p">:</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="n">d</span> <span class="o">=</span> <span class="n">Door</span><span class="p">()</span>
<span class="gp">>>> </span><span class="nb">isinstance</span><span class="p">(</span><span class="n">d</span><span class="p">,</span> <span class="n">Door</span><span class="p">)</span>
<span class="go">True</span>
<span class="gp">>>> </span><span class="k">class</span> <span class="nc">EnhancedDoor</span><span class="p">(</span><span class="n">Door</span><span class="p">):</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="n">ed</span> <span class="o">=</span> <span class="n">EnhancedDoor</span><span class="p">()</span>
<span class="gp">>>> </span><span class="nb">isinstance</span><span class="p">(</span><span class="n">ed</span><span class="p">,</span> <span class="n">EnhancedDoor</span><span class="p">)</span>
<span class="go">True</span>
<span class="gp">>>> </span><span class="nb">isinstance</span><span class="p">(</span><span class="n">ed</span><span class="p">,</span> <span class="n">Door</span><span class="p">)</span>
<span class="go">True</span>
</code></pre></div>
<p>As you can see the function can also walk the class hierarchy, so the check is not so trivial like the one you would obtain by directly using <code>type()</code>.</p>
<p>The <code>isinstance()</code> function, however, does not completely solve the problem. If we write a class that actually <em>acts</em> like a <code>list</code> but does not inherit from it, <code>isinstance()</code> does not recognize the fact that the two may be considered the same thing. The following code returns <code>False</code> regardless the content of the <code>MyList</code> class</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="k">class</span> <span class="nc">MyList</span><span class="p">:</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="n">ml</span> <span class="o">=</span> <span class="n">MyList</span><span class="p">()</span>
<span class="gp">>>> </span><span class="nb">isinstance</span><span class="p">(</span><span class="n">ml</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span>
<span class="go">False</span>
</code></pre></div>
<p>since <code>isinstance()</code> does not check the content of the class or its behaviour, it just considers the class and its ancestors.</p>
<p>The problem, thus, may be summed up with the following question: what is the best way to test that an object exposes a given interface? Here, the word <em>interface</em> is used for its natural meaning, without any reference to other programming solutions, which however address the same problem.</p>
<p>A good way to address the problem could be to write inside an attribute of the object the list of interfaces it promises to implement, and to agree that any time we want to test the behaviour of an object we simply have to check the content of this attribute. This is exactly the path followed by Python, and it is very important to understand that the whole system is just about a promised behaviour.</p>
<p>The solution proposed through PEP 3119 is, in my opinion, very simple and elegant, and it perfectly fits the nature of Python, where things are usually agreed rather than being enforced. Not only, the solution follows the spirit of polymorphism, where information is provided by the object itself and not extracted by the calling code.</p>
<p>In the next sections I am going to try and describe this solution in its main building blocks. The matter is complex so my explanation will lack some details: please refer to the forementioned PEP 3119 for a complete description.</p>
<h4 id="who-framed-the-metaclasses">Who Framed the Metaclasses<a class="headerlink" href="#who-framed-the-metaclasses" title="Permanent link">¶</a></h4>
<p>As already described, Python provides two built-ins to inspect objects and classes, which are <code>isinstance()</code> and <code>issubclass()</code> and a solution to the inspection problem should allow the programmer to go on with using those two functions.</p>
<p>This means that we need to find a way to inject the "behaviour promise" into both classes and instances. This is the reason why metaclasses come in play. Recall what we said about them in the fifth issue of this series: metaclasses are the classes used to build classes, which means that they are the preferred way to change the structure of a class, and, in consequence, of its instances.</p>
<p>Another way to do the same job would be to leverage the inheritance mechanism, injecting the behaviour through a dedicated parent class. This solution has many downsides, which I'm am not going to detail. It is enough to say that affecting the class hierarchy may lead to complex situations or subtle bugs. Metaclasses may provide here a different entry point for the introduction of a "virtual base class" (as PEP 3119 specifies, this is not the same concept as in C++).</p>
<h4 id="overriding-places">Overriding Places<a class="headerlink" href="#overriding-places" title="Permanent link">¶</a></h4>
<p>As said, <code>isinstance()</code> and <code>issubclass()</code> are built-in functions, not object methods, so we cannot simply override them providing a different implementation in a given class. So the first part of the solution is to change the behaviour of those two functions to first check if the class or the instance contain a special method, which is <code>__instancecheck__()</code> for <code>isinstance()</code> and <code>__subclasscheck__()</code> for <code>issubclass()</code>. So both built-ins try to run the respective special method, reverting to the standard algorithm if it is not present.</p>
<p>A note about naming. Methods must accept the object they belong to as the first argument, so the two special methods shall have the form</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="fm">__instancecheck__</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">inst</span><span class="p">):</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="k">def</span> <span class="fm">__subclasscheck__</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">sub</span><span class="p">):</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
</code></pre></div>
<p>where <code>cls</code> is the class where they are injected, that is the one representing the promised behaviour. The two built-ins, however, have a reversed argument order, where the behaviour comes after the tested object: when you write <code>isinstance([], list)</code> you want to check if the <code>[]</code> instance has the <code>list</code> behaviour. This is the reason behind the name choice: just calling the methods <code>__isinstance__()</code> and <code>__issubclass__()</code> and passing arguments in a reversed order would have been confusing.</p>
<h4 id="this-is-abc">This is ABC<a class="headerlink" href="#this-is-abc" title="Permanent link">¶</a></h4>
<p>The proposed solution is thus called Abstract Base Classes, as it provides a way to attach to a concrete class a virtual class with the only purpose of signaling a promised behaviour to anyone inspecting it with <code>isinstance()</code> or <code>issubclass()</code>.</p>
<p>To help programmers implement Abstract Base Classes, the standard library has been given an <code>abc</code> module, thet contains the <code>ABCMeta</code> class (and other facilities). This class is the one that implements <code>__instancecheck__()</code> and <code>__subclasscheck__()</code> and shall be used as a metaclass to augment a standard class. The latter will then be able to register other classes as implementation of its behaviour.</p>
<p>Sounds complex? An example may clarify the whole matter. The one from the official documentation is rather simple:</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">abc</span> <span class="kn">import</span> <span class="n">ABCMeta</span>
<span class="k">class</span> <span class="nc">MyABC</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">ABCMeta</span><span class="p">):</span>
<span class="k">pass</span>
<span class="n">MyABC</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="nb">tuple</span><span class="p">)</span>
<span class="k">assert</span> <span class="nb">issubclass</span><span class="p">(</span><span class="nb">tuple</span><span class="p">,</span> <span class="n">MyABC</span><span class="p">)</span>
<span class="k">assert</span> <span class="nb">isinstance</span><span class="p">((),</span> <span class="n">MyABC</span><span class="p">)</span>
</code></pre></div>
<p>Here, the <code>MyABC</code> class is provided the <code>ABCMeta</code> metaclass. This puts the two <code>__instancecheck__()</code> and <code>__subclasscheck__()</code> methods inside <code>MyABC</code> so that, when issuing <code>isinstance()</code>, what Python actually ececutes is</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">d</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'a'</span><span class="p">:</span> <span class="mi">1</span><span class="p">}</span>
<span class="gp">>>> </span><span class="nb">isinstance</span><span class="p">(</span><span class="n">d</span><span class="p">,</span> <span class="n">MyABC</span><span class="p">)</span>
<span class="go">False</span>
<span class="gp">>>> </span><span class="n">MyABC</span><span class="o">.</span><span class="vm">__class__</span><span class="o">.</span><span class="fm">__instancecheck__</span><span class="p">(</span><span class="n">MyABC</span><span class="p">,</span> <span class="n">d</span><span class="p">)</span>
<span class="go">False</span>
<span class="gp">>>> </span><span class="nb">isinstance</span><span class="p">((),</span> <span class="n">MyABC</span><span class="p">)</span>
<span class="go">True</span>
<span class="gp">>>> </span><span class="n">MyABC</span><span class="o">.</span><span class="vm">__class__</span><span class="o">.</span><span class="fm">__instancecheck__</span><span class="p">(</span><span class="n">MyABC</span><span class="p">,</span> <span class="p">())</span>
<span class="go">True</span>
</code></pre></div>
<p>After the definition of <code>MyABC</code> we need a way to signal that a given class is an instance of the Abstract Base Class and this happens through the <code>register()</code> method, provided by the <code>ABCMeta</code> metaclass. Calling <code>MyABC.register(tuple)</code> we record inside <code>MyABC</code> the fact that the <code>tuple</code> class shall be identified as a subclass of <code>MyABC</code> itself. This is analogous to saying that <code>tuple</code> inherits from <code>MyABC</code> but not quite the same. As already said registering a class in an Abstract Base Class with <code>register()</code> does not affect the class hierarchy. Indeed, the whole <code>tuple</code> class is unchanged.</p>
<p>The current implementation of ABCs stores the registered types inside the <code>_abc_registry</code> attribute. Actually it stores there weak references to the registered types (this part is outside the scope of this article, so I'm not detailing it)</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">MyABC</span><span class="o">.</span><span class="n">_abc_registry</span><span class="o">.</span><span class="n">data</span>
<span class="go">{<weakref at 0xb682966c; to 'type' at 0x83dcca0 (tuple)>}</span>
</code></pre></div>
<h2 id="movie-trivia">Movie Trivia<a class="headerlink" href="#movie-trivia" title="Permanent link">¶</a></h2>
<p>Section titles come from the following movies: <em>The Breakfast Club (1985)</em>, <em>E.T. the Extra-Terrestrial (1982)</em>, <em>Who Framed Roger Rabbit (1988)</em>, <em>Trading Places (1983)</em>, <em>This is Spinal Tap (1984)</em>.</p>
<h2 id="sources">Sources<a class="headerlink" href="#sources" title="Permanent link">¶</a></h2>
<p>You will find a lot of documentation in <a href="https://www.reddit.com/r/Python/comments/226ahl/some_links_about_python_oop/">this Reddit post</a>. Most of the information contained in this series come from those sources.</p>
<h2 id="feedback">Feedback<a class="headerlink" href="#feedback" title="Permanent link">¶</a></h2>
<p>The <a href="https://github.com/TheDigitalCatOnline/blog_source/issues">GitHub issues</a> page is the best place to submit corrections.</p>Object-Oriented Programming in Python 3 - Metaclasses2014-09-01T15:00:00+02:002019-05-22T23:30:00+00:00Leonardo Giordanitag:www.thedigitalcatonline.com,2014-09-01:/blog/2014/09/01/python-3-oop-part-5-metaclasses/<p>Fundamentals of object-oriented programming in Python - metaclasses</p><p>This post is available as an <strong>IPython Notebook</strong> <a href="/notebooks/Python_3_OOP_Part_5__Metaclasses.ipynb">here</a></p>
<h2 id="the-type-brothers">The Type Brothers<a class="headerlink" href="#the-type-brothers" title="Permanent link">¶</a></h2>
<p>The first step into the most intimate secrets of Python objects comes from two components we already met in the first post: <code>class</code> and <code>object</code>. These two things are the very fundamental elements of Python OOP system, so it is worth spending some time to understand how they work and relate each other.</p>
<p>First of all recall that in Python <em>everything is an object</em>, that is everything inherits from <code>object</code>. Thus, <code>object</code> seems to be the deepest thing you can find digging into Python variables. Let's check this</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">a</span> <span class="o">=</span> <span class="mi">5</span>
<span class="gp">>>> </span><span class="nb">type</span><span class="p">(</span><span class="n">a</span><span class="p">)</span>
<span class="go"><class 'int'></span>
<span class="gp">>>> </span><span class="n">a</span><span class="o">.</span><span class="vm">__class__</span>
<span class="go"><class 'int'></span>
<span class="gp">>>> </span><span class="n">a</span><span class="o">.</span><span class="vm">__class__</span><span class="o">.</span><span class="vm">__bases__</span>
<span class="go">(<class 'object'>,)</span>
<span class="gp">>>> </span><span class="nb">object</span><span class="o">.</span><span class="vm">__bases__</span>
<span class="go">()</span>
</code></pre></div>
<p>The variable <code>a</code> is an instance of the <code>int</code> class, and the latter inherits from <code>object</code>, which inherits from nothing. This demonstrates that <code>object</code> is at the top of the class hierarchy. However, as you can see, both <code>int</code> and <code>object</code> are called <em>classes</em> (<code><class 'int'></code>, <code><class 'object'></code>). Indeed, while <code>a</code> is an instance of the <code>int</code> class, <code>int</code> itself is an instance of another class, <em>a class that is instanced to build classes</em></p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="nb">type</span><span class="p">(</span><span class="n">a</span><span class="p">)</span>
<span class="go"><class 'int'></span>
<span class="gp">>>> </span><span class="nb">type</span><span class="p">(</span><span class="nb">int</span><span class="p">)</span>
<span class="go"><class 'type'></span>
<span class="gp">>>> </span><span class="nb">type</span><span class="p">(</span><span class="nb">float</span><span class="p">)</span>
<span class="go"><class 'type'></span>
<span class="gp">>>> </span><span class="nb">type</span><span class="p">(</span><span class="nb">dict</span><span class="p">)</span>
<span class="go"><class 'type'></span>
</code></pre></div>
<p>Since in Python everything is an object, everything is the instance of a class, even classes. Well, <code>type</code> is the class that is instanced to get classes. So remember this: <code>object</code> is the base of every object, <code>type</code> is the class of every type. Sounds puzzling? It is not your fault, don't worry. However, just to strike you with the finishing move, this is what Python is built on</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="nb">type</span><span class="p">(</span><span class="nb">object</span><span class="p">)</span>
<span class="go"><class 'type'></span>
<span class="gp">>>> </span><span class="nb">type</span><span class="o">.</span><span class="vm">__bases__</span>
<span class="go">(<class 'object'>,)</span>
</code></pre></div>
<p>If you are not about to faint at this point chances are that you are Guido van Rossum or one of his friends down at the Python core development team (in this case let me thank you for your beautiful creation). You may get a cup of tea, if you need it.</p>
<p>Jokes apart, at the very base of Python type system there are two things, <code>object</code> and <code>type</code>, which are inseparable. The previous code shows that <code>object</code> is an instance of <code>type</code>, and <code>type</code> inherits from <code>object</code>. Take your time to understand this subtle concept, as it is very important for the upcoming discussion about metaclasses.</p>
<p>When you think you grasped the <code>type</code>/<code>object</code> matter read this and start thinking again</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="nb">type</span><span class="p">(</span><span class="nb">type</span><span class="p">)</span>
<span class="go"><class 'type'></span>
</code></pre></div>
<h2 id="the-metaclasses-take-python">The Metaclasses Take Python<a class="headerlink" href="#the-metaclasses-take-python" title="Permanent link">¶</a></h2>
<p>You are now familiar with Python classes. You know that a class is used to create an instance, and that the structure of the latter is ruled by the source class and all its parent classes (until you reach <code>object</code>).</p>
<p>Since classes are objects too, you know that a class itself is an instance of a (super)class, and this class is <code>type</code>. That is, as already stated, <code>type</code> is the class that is used to build classes.</p>
<p>So for example you know that a class may be instanced, i.e. it can be called and by calling it you obtain another object that is linked with the class. What prepares the class for being called? What gives the class all its methods? In Python the class in charge of performing such tasks is called <em>metaclass</em>, and <code>type</code> is the default metaclass of all classes.</p>
<p>The point of exposing this structure of Python objects is that you may change the way classes are built. As you know, <code>type</code> is an object, so it can be subclassed just like any other class. Once you get a subclass of <code>type</code> you need to instruct your class to use it as the metaclass instead of type, and you can do this by passing it as the <code>metaclass</code> keyword argument in the class definition.</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="k">class</span> <span class="nc">MyType</span><span class="p">(</span><span class="nb">type</span><span class="p">):</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">...</span>
<span class="gp">>>> </span><span class="k">class</span> <span class="nc">MySpecialClass</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">MyType</span><span class="p">):</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="n">msp</span> <span class="o">=</span> <span class="n">MySpecialClass</span><span class="p">()</span>
<span class="gp">>>> </span><span class="nb">type</span><span class="p">(</span><span class="n">msp</span><span class="p">)</span>
<span class="go"><class '__main__.MySpecialClass'></span>
<span class="gp">>>> </span><span class="nb">type</span><span class="p">(</span><span class="n">MySpecialClass</span><span class="p">)</span>
<span class="go"><class '__main__.MyType'></span>
<span class="gp">>>> </span><span class="nb">type</span><span class="p">(</span><span class="n">MyType</span><span class="p">)</span>
<span class="go"><class 'type'></span>
</code></pre></div>
<h4 id="metaclasses-2-singleton-day">Metaclasses 2: Singleton Day<a class="headerlink" href="#metaclasses-2-singleton-day" title="Permanent link">¶</a></h4>
<p>Metaclasses are a very advanced topic in Python, but they have many practical uses. For example, by means of a custom metaclass you may log any time a class is instanced, which can be important for applications that shall keep a low memory usage or have to monitor it.</p>
<p>I am going to show here a very simple example of metaclass, the Singleton. Singleton is a well known design pattern, and many description of it may be found on the Internet. It has also been heavily criticized mostly because its bad behaviour when subclassed, but here I do not want to introduce it for its technological value, but for its simplicity (so please do not question the choice, it is just an example).</p>
<p>Singleton has one purpose: to return the same instance every time it is instanced, like a sort of object-oriented global variable. So we need to build a class that does not work like standard classes, which return a new instance every time they are called.</p>
<p>"Build a class"? This is a task for metaclasses. The following implementation comes from <a href="http://python-3-patterns-idioms-test.readthedocs.org/en/latest/Metaprogramming.html#intercepting-class-creation">Python 3 Patterns, Recipes and Idioms</a>.</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">Singleton</span><span class="p">(</span><span class="nb">type</span><span class="p">):</span>
<span class="n">instance</span> <span class="o">=</span> <span class="kc">None</span>
<span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kw</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="bp">cls</span><span class="o">.</span><span class="n">instance</span><span class="p">:</span>
<span class="bp">cls</span><span class="o">.</span><span class="n">instance</span> <span class="o">=</span> <span class="nb">super</span><span class="p">(</span><span class="n">Singleton</span><span class="p">,</span> <span class="bp">cls</span><span class="p">)</span><span class="o">.</span><span class="fm">__call__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kw</span><span class="p">)</span>
<span class="k">return</span> <span class="bp">cls</span><span class="o">.</span><span class="n">instance</span>
</code></pre></div>
<p>We are defining a new type, which inherits from <code>type</code> to provide all bells and whistles of Python classes. We override the <code>__call__</code> method, that is a special method invoked when we call the class, i.e. when we instance it. The new method wraps the original method of <code>type</code> by calling it only when the <code>instance</code> attribute is not set, i.e. the first time the class is instanced, otherwise it just returns the recorded instance. As you can see this is a very basic cache class, the only trick is that it is applied to the creation of instances.</p>
<p>To test the new type we need to define a new class that uses it as its metaclass</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="k">class</span> <span class="nc">ASingleton</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">Singleton</span><span class="p">):</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="n">a</span> <span class="o">=</span> <span class="n">ASingleton</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">b</span> <span class="o">=</span> <span class="n">ASingleton</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">a</span> <span class="ow">is</span> <span class="n">b</span>
<span class="go">True</span>
<span class="gp">>>> </span><span class="nb">hex</span><span class="p">(</span><span class="nb">id</span><span class="p">(</span><span class="n">a</span><span class="p">))</span>
<span class="go">'0xb68030ec'</span>
<span class="gp">>>> </span><span class="nb">hex</span><span class="p">(</span><span class="nb">id</span><span class="p">(</span><span class="n">b</span><span class="p">))</span>
<span class="go">'0xb68030ec'</span>
</code></pre></div>
<p>By using the <code>is</code> operator we test that the two objects are the very same structure in memory, that is their ids are the same, as explicitly shown. What actually happens is that when you issue <code>a = ASingleton()</code> the <code>ASingleton</code> class runs its <code>__call__()</code> method, which is taken from the <code>Singleton</code> type behind the class. That method recognizes that no instance has been created (<code>Singleton.instance</code> is <code>None</code>) and acts just like any standard class does. When you issue <code>b = ASingleton()</code> the very same things happen, but since <code>Singleton.instance</code> is now different from <code>None</code> its value (the previous instance) is directly returned.</p>
<p>Metaclasses are a very powerful programming tool and leveraging them you can achieve very complex behaviours with a small effort. Their use is a must every time you are actually metaprogramming, that is if you are writing code that has to drive the way your code works. Good examples are creational patterns (injecting custom class attributes depending on some configuration), testing, debugging, and performance monitoring.</p>
<h2 id="coming-to-instance">Coming to Instance<a class="headerlink" href="#coming-to-instance" title="Permanent link">¶</a></h2>
<p>Before introducing you to a very smart use of metaclasses by talking about Abstract Base Classes (read: to save some topics for the next part of this series), I want to dive into the object creation procedure in Python, that is what happens when you instance a class. In the first post this procedure was described only partially, by looking at the <code>__init__()</code> method.</p>
<p>In the first post I recalled the object-oriented concept of <em>constructor</em>, which is a special method of the class that is automatically called when the instance is created. The class may also define a destructor, which is called when the object is destroyed. In languages without a garbage collection mechanism such as C++ the destructor shall be carefully designed. In Python the destructor may be defined through the <code>__del__()</code> method, but it is hardly used.</p>
<p>The constructor mechanism in Python is on the contrary very important, and it is implemented by two methods, instead of just one: <code>__new__()</code> and <code>__init__()</code>. The tasks of the two methods are very clear and distinct: <code>__new__()</code> shall perform actions needed when <em>creating</em> a new instance while <code>__init__</code> deals with object <em>initialization</em>.</p>
<p>Since in Python you do not need to declare attributes due to its dynamic nature, <code>__new__()</code> is rarely defined by programmers, who may rely on <code>__init__</code> to perform the majority of the usual tasks. Typical uses of <code>__new__()</code> are very similar to those listed in the previous section, since it allows to trigger some code whenever your class is instanced.</p>
<p>The standard way to override <code>__new__()</code> is </p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">MyClass</span><span class="p">():</span>
<span class="k">def</span> <span class="fm">__new__</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwds</span><span class="p">):</span>
<span class="n">obj</span> <span class="o">=</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__new__</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwds</span><span class="p">)</span>
<span class="p">[</span><span class="n">put</span> <span class="n">your</span> <span class="n">code</span> <span class="n">here</span><span class="p">]</span>
<span class="k">return</span> <span class="n">obj</span>
</code></pre></div>
<p>just like you usually do with <code>__init__()</code>. When your class inherits from <code>object</code> you do not need to call the parent method (<code>object.__init__()</code>), because it is empty, but you need to do it when overriding <code>__new__</code>.</p>
<p>Remember that <code>__new__()</code> is not forced to return an instance of the class in which it is defined, even if you shall have very good reasons to break this behaviour. Anyway, <code>__init__()</code> will be called only if you return an instance of the container class. Please also note that <code>__new__()</code>, unlike <code>__init__()</code>, accepts the class as its first parameter. The name is not important in Python, and you can also call it <code>self</code>, but it is worth using <code>cls</code> to remember that it is not an instance.</p>
<h2 id="movie-trivia">Movie Trivia<a class="headerlink" href="#movie-trivia" title="Permanent link">¶</a></h2>
<p>Section titles come from the following movies: <em>The Blues Brothers (1980)</em>, <em>The Muppets Take Manhattan (1984)</em>, <em>Terminator 2: Judgement Day (1991)</em>, <em>Coming to America (1988)</em>.</p>
<h2 id="sources">Sources<a class="headerlink" href="#sources" title="Permanent link">¶</a></h2>
<p>You will find a lot of documentation in <a href="https://www.reddit.com/r/Python/comments/226ahl/some_links_about_python_oop/">this Reddit post</a>. Most of the information contained in this series come from those sources.</p>
<h2 id="feedback">Feedback<a class="headerlink" href="#feedback" title="Permanent link">¶</a></h2>
<p>The <a href="https://github.com/TheDigitalCatOnline/blog_source/issues">GitHub issues</a> page is the best place to submit corrections.</p>