Another Python metaclass primer

metapoop.py

#! /usr/bin/env python

"""
Python metaclasses
==================

A metaclass is a class factory; metaclasses serve two purposes:

1. replace ``type`` as the base class metatype for classes with the
   ``__metaclass__`` attribute
2. act as a class factory, to create classes dynamically

references
----------
1. StackOverflow answer to `What is a metaclass in Python? \
   <http://stackoverflow.com/a/6581949/1020470>`_
2. Eli Bendersky's website on `Python metaclasses by example \
   <http://eli.thegreenplace.net/2011/08/14/python-metaclasses-by-example/>`_
3. `Python Built-in Functions <http://docs.python.org/2/library/functions.html#type>`_
4. `Overriding the __new__ method \
   <http://www.python.org/download/releases/2.2/descrintro/#__new__>`_ and
   'metaclasses <http://www.python.org/download/releases/2.2/descrintro/#metaclasses>`_
5. `A Primer on Python Metaclass Programming \
   <http://www.onlamp.com/pub/a/python/2003/04/17/metaclasses.html>`_
6. `Python Data Model \
   <http://docs.python.org/2/reference/datamodel.html#customizing-class-creation>`_
7. `A Primer on Python Metaclasses \
   <http://jakevdp.github.io/blog/2012/12/01/a-primer-on-python-metaclasses/>`_
8. Effbot `metaclass <http://effbot.org/pyref/method-metaclass.htm>`_
   `attributes <http://effbot.org/pyref/__metaclass__.htm>`_
9. `Magic Methods <http://www.rafekettler.com/magicmethods.html>`_
10. `Python Idioms \
    <http://python-3-patterns-idioms-test.readthedocs.org/en/latest/Metaprogramming.html>`_

type
----
``type`` is both a function and a class depending on the interface. As a
function it returns the type of the object passed as its only argument. It is
the base class for all Python objects and is its own type.

    >>> type(type)
    type
    >>> type(object)
    type
    >>> type(int)
    type
    
``type`` is also a class factory when passed the name <str>, tuple of bases
and dictionary of class attributes. Note that functions that are passed as
attributes must call a class instance as its first argument or use the
``@classmethod`` or ``@staticmethod`` decorators. Unbound methods can not be
called.

    >>> MyClass = type('MyClass', (object, ),
    ...                {'cls_attr': 'foobar',
    ...                 'inst_meth': 'lambda self, x: x**2})
    >>> print MyClass
    <class '__main__.MyClass'>
    >>> print MyClass.cls_attr
    foobar
    >>> mc = MyClass()
    >>> print mc
    <__main__.MyClass object at 0x0000000003F85748>
    >>> print mc.inst_meth(3)
    9

``__metaclass__`` attribute
---------------------------
If a class has the ``__metaclass__`` attribute set to a callable that returns
a class, it will use that metaclass to create the class. If it doesn't find a
``__metaclass__`` attribute it will look in any superclasses and finally use
``type``. Either a function that calls ``type()`` or a subclass of ``type`` can
be the ``__metaclass__`` attribute. A metaclass can be used to change bases or
class attributes before they class object is created.

Class Factory
-------------
A subclass of ``type()`` can be used to alter the bases or class attributes of
classes before they're created or as a class factory to create classes
dynamically.

``__new__`` and ``__init__`` methods
------------------------------------
All classes, including metaclasses first call ``__new__`` to create the class
object followed by ``__init__`` which instantiates the class object. The
most important differences between ``__new__`` and ``__init__`` are

1. ``__new__`` is always called first, immediately followed by ``__init__``
2. ``__new__`` must return the newly instantieated class object, but ``__init``
   doesn`t return anything, it only intiallizes the instance.

The ``__new__`` method is mainly used for subclassing immutable types. For
example use the ``__new__`` method to subclass a ``numpy.ndarray`` and return
a view of the argument as an array.

In metaclasses inputs to ``type()`` can be altered in the ``__new__`` method
before returning the new class factory, but in the ``__init__`` method changes
will have **no** effect on the class factory, because ``__init___`` doesn't
return anything. However, the class factory instance can be altered in the
``__init__`` method, since it is used to initialize the instance. EG: new
attributes can be assigned directly to the class factory instance using dot
notation. In the ``__new__`` method, assigning attributes directly to the new
class factory object using dot notation has the same effect of creating class
variables in the metaclass.
"""

def print_cls_name_bases_attr(cls, name, bases, attr):
    """
    print the class, name, bases tuple, and dictionary of attributes
    """
    print 'class:', cls
    print 'name:', name
    print 'bases:', bases
    print 'attr:', attr
    print '------------------------'


class Meta(type):
    FOO = 'hi'
    def __new__(cls, name, bases, attr):
        # cls.FOO = 'hi'  # equivalent assignment of class attribute
        # change args before returning new class factory, EG: add a class
        # attribute `foo` equal to the meta class constant `FOO`
        if 'bar' in attr:
            attr['foo'] = Meta.FOO
        print 'in Meta __new__ method:'
        print_cls_name_bases_attr(cls, name, bases, attr)
        return super(Meta, cls).__new__(cls, name, bases, attr)
    def __init__(cls, name, bases, attr):
        # changing name, bases or attr in __init__ has no effect!
        # attr['this'] = 'does nothing'
        super(Meta,cls).__init__(name, bases, attr)
        # change class directly, it was already created in `__new__`
        if 'foo' in attr:
            cls.barfoo = 2000
        print 'in Meta __init__ method:'
        print_cls_name_bases_attr(cls, name, bases, attr)


class FooBar(object):
    __metaclass__ = Meta  # use Meta instead of type
    bar = 1999
    def __init__(self, x):
        self.x = 'x is %s, BAR is %s' % (x, FooBar.bar)


print 'FooBar dict:', FooBar.__dict__
f = FooBar('bye')
print 'x:', f.x, '\nbar:', FooBar.bar, '\nfoo:', FooBar.foo, '\nbarfoo:', FooBar.barfoo
print 'f dict:', f.__dict__
print '----------------------'

"""
As Python reads the module it looks at the interface of each function or class
and creates an object in memory for it. When it comes across a class with the
``__metaclass__`` attribute, it parses the class definition for the "name",
"bases" and "attributes" and executes the callable given by the
``__metaclass__`` immediately. This is why you see the output from the meta
classes first.

    in Meta __new__ method:
    class: <class '__main__.Meta'>
    name: FooBar
    bases: (<type 'object'>,)
    attr: {
        'bar': 1999,
        '__module__': '__main__',
        'foo': 'hi',
        '__metaclass__': <class '__main__.Meta'>,
        '__init__': <function __init__ at 0x0000000002327D68>}
    ------------------------
    in Meta __init__ method:
    class: <class '__main__.FooBar'>
    name: FooBar
    bases: (<type 'object'>,)
    attr: {
        'bar': 1999,
        '__module__': '__main__',
        'foo': 'hi',
        '__metaclass__': <class '__main__.Meta'>,
        '__init__': <function __init__ at 0x0000000002327D68>}
    ------------------------
    FooBar dict: {
        '__module__': '__main__',
        '__metaclass__': <class '__main__.Meta'>,
        'barfoo': 2000,
        '__dict__': <attribute '__dict__' of 'FooBar' objects>,
        'bar': 1999,
        'foo': 'hi',
        '__weakref__': <attribute '__weakref__' of 'FooBar' objects>,
        '__doc__': None,
        '__init__': <function __init__ at 0x0000000002327D68>}
    x: x is bye, BAR is 1999
    bar: 1999
    foo: hi
    barfoo: 2000
    f dict: {'x': 'x is bye, BAR is 1999'}
"""

import os, json

_DIRNAME = os.path.dirname(__file__)
_OUTPUTS = os.path.join(_DIRNAME, '.')


class OutputSources(object):
    def __init__(self, param_file):
        self.param_file = param_file
        with open(param_file, 'r') as fp:
            #: parameters from file for outputs
            self.parameters = json.load(fp)

"""
Now we'll define a metaclass that adds a desired subclass if it is missing,
which must be done in ``__new__`` before the metaclass is created. We'll also
create a constructor and pass it as the ``__init__`` attribute of the new
as yet uncreated class; not to be confused with the ``__init__`` method of the
metaclass. We'll also pass an extra argument in the metaclass constructor
to set instance attributes inside the as yet uncreated class `__init__` method.

Note as in the example on using just ``type()`` to dynamically create a class,
when passing methods as attributes the first argument must be the class instance,
not the class, unless the function is decorated with ``@classmethod`` or
``@staticmethod``.

You can pass extra args to ``__new__``, and still call ``super`` for ``type()``
with the correct interface. Note that ``type()`` requires 1 or 3 inputs only.
But weird things can happen in ``__init__`` because it will receive the exact
same arguments as ``__new__``, so if we want to pass extra arguments to the
metaclass, then we will need to override the metaclass ``__init__`` method and
pass it exactly the same args that you pass ``__new__``. This is true for any
class constructor, not just metaclasses.

When creating the metaclass if you set defaults args for ``__init__`` weird
things can happen. Whatever you set as the default for the class attributes,
will "appear" to overwrite whatever attributes where passed from ``__new__``,
but this has **no** effect because the class attributes have already been
created in ``__new__`` and the arguments passed to ``__init__`` have no effect.

Another example of this is the ``bases`` arg which is not passed from ``__new__``
to ``__init__``, which doesn't matter, because any base classes are already set
in ``__new__``. So if you print the args in ``__init__`` you will see only
the bases passed as args in the call to create the metaclass, and any bases
added to the class in ``__new__`` don't appear. However, all of the attributes
set in ``__new__`` do get passed to ``__init__``. Practially what this means is
that you should only use ``__new__`` to make changes based on what the bases are,
but you *can* use ``__init__`` to make changes based on class attributes.

Compare this with how metaclass is used when its arguments come from the
``__metaclass__`` attribute. When you instantiate the class directly from
the metaclass, you must provide all of the arguments that eventually get passed
to ``type()``, but when the ``__metaclass__`` attribute is used, it will
collect the arguments by parsing the class definition. In both cases, exactly
the same args get passed to both ``__new__`` and ``__init__`` as always, and
just like creating a new class from a metaclass by directly calling it, bases
added in ``__new__`` do not get passed to ``__init__``.

Note also that since this class is created dynamically, the metaclass isn't
instantiated until it's actually called in the __main__ section of the script.
However, the constructor inside the metaclass still doesn't run until the
dynamically created class is instantiated.

    cls is <class '__main__.MetaOutputSource'>
    __dict__ is {
        '__module__': '__main__',
        '__new__': <staticmethod object at 0x00000000023810D8>,
        '__init__': <function __init__ at 0x0000000002384D68>,
        '__doc__': None}
    __class__ is <type 'type'>
    __class__.__class__ is <type 'type'>
    __class__.__bases__ are (<type 'object'>,)
    ----------------------
    in MetaOutputSource __new__ method
    class: <class '__main__.MetaOutputSource'>
    name: PoopOut
    bases: (<class '__main__.OutputSources'>,)
    attr: {'__init__': <function __init__ at 0x0000000002384F28>}
    ------------------------
    in MetaOutputSource __init__ method
    class: <class '__main__.PoopOut'>
    name: PoopOut
    bases: ()
    attr: {'__init__': <function __init__ at 0x0000000002384F28>}
    ------------------------
    cls is <class '__main__.PoopOut'>
    __dict__ is {
        '__module__': '__main__',
        '__doc__': None,
        '__init__': <function __init__ at 0x0000000002384F28>}
    __class__ is <class '__main__.MetaOutputSource'>
    __class__.__class__ is <type 'type'>
    __class__.__bases__ are (<type 'type'>,)
    ----------------------
    inside constructor for dynamically created classes
    set Cool_Fx to 100
    ----------------------
    cls is <__main__.PoopOut object at 0x0000000002382AC8>
    __dict__ is {
        'cool_Fx': 100,
        'parameters': {
            u'poop': u'crap',
            u'foo': [1, 2, 3],
            u'bar': {u'this': u'that'}},
        'param_file': '.\\.\\poop.json'}
    __class__ is <class '__main__.PoopOut'>
    __class__.__class__ is <class '__main__.MetaOutputSource'>
    __class__.__bases__ are (<class '__main__.OutputSources'>,)
    ----------------------
    cls is <class '__main__.OutputSources'>
    __dict__ is {
        '__dict__': <attribute '__dict__' of 'OutputSources' objects>,
        '__module__': '__main__',
        '__weakref__': <attribute '__weakref__' of 'OutputSources' objects>,
        '__doc__': None,
        '__init__': <function __init__ at 0x0000000002384C88>}
    __class__ is <type 'type'>
    __class__.__class__ is <type 'type'>
    __class__.__bases__ are (<type 'object'>,)
    ----------------------
    cls is <__main__.OutputSources object at 0x0000000002382BA8>
    __dict__ is {
        'parameters': {
            u'poop': u'crap',
            u'foo': [1, 2, 3],
            u'bar': {u'this': u'that'}},
        'param_file': 'poop.json'}
    __class__ is <class '__main__.OutputSources'>
    __class__.__class__ is <type 'type'>
    __class__.__bases__ are (<type 'object'>,)
"""

            
class MetaOutputSource(type):
    def __new__(cls, name, bases, attr):
        # we can change the name, base and attributes of the class to be
        # created here in the `__new__` method.
        # we *could* also pass extra args to the metaclass if we are going to
        # call it directly to make our class object first, but not if we're
        # going to add it as a `__metaclass__` attribute. See `__call__`.
        
        # we'll check the bases and add our desired base if it's missing,
        # since we're going to override that base's `__init__` method.
        if OutputSources not in bases:
            bases += (OutputSources, )  # must subclass OutputSources

        def __init__(self, param_file, inst_attr={'dumb_Fx': 50}):
            # this doesn't get called until after both the class object and
            # metaclass instances have been created!
            # So there is no way to remove the class attributes and set them
            # as instance attributes only
            param_file = os.path.join(_OUTPUTS, param_file)
            print 'inside constructor for dynamically created classes'
            for k, v in inst_attr.iteritems():
                setattr(self, k, v)
                print "set %s to %s" % (k, v)
            print '----------------------'
            OutputSources.__init__(self, param_file)

        attr.update({'__init__': __init__})  # add constructor as attribute
        print 'in MetaOutputSource __new__ method'
        print_cls_name_bases_attr(cls,name,bases,attr)
        return super(MetaOutputSource, cls).__new__(cls, name, bases, attr)

    def __init__(cls, name, bases, attr):
        # if OutputSources not in bases:
            # bases += (dict, )  # this has **no** effect
        # Can't add to attributes here, because class object is already created
        # even though it hasn't been instantiated
        super(MetaOutputSource, cls).__init__(name, bases, attr)
        print 'in MetaOutputSource __init__ method'
        print_cls_name_bases_attr(cls, name, bases, attr)

"""
This next example is actually a repeat of the 1st ``__metaclass__`` attribute
example. Python goes through the module looking for definitions, allocating
memory. When it sees a class with a ``__metaclass__`` attribute it immediately
starts creating that class factory, using the metaclass, but not the actual
class onject. This use of metaclasses is a bit like a decorator or subclassing
however it does have the advantage that the metaclass type is only created
once, whereas a decorated or subclassed class definition would be recreated
every single time.

    in MetaPoop __new__ method
    class: <class '__main__.MetaPoop'>
    name: PoopSources
    bases: (<class '__main__.OutputSources'>,
            <class __main__.St00pid at 0x0000000002364B28>)
    attr: {
        '__module__': '__main__',
        '__metaclass__': <class '__main__.MetaPoop'>,
        'PARAM_FILE': '.\\.\\poop.json',
        'POOPY': 'poopy.json',
        'POOP': 'poop.json',
        '__init__': <function __init__ at 0x0000000002384EB8>}
    ------------------------
    in MetaPoop __init__ method
    class: <class '__main__.PoopSources'>
    name: PoopSources
    bases: (<class '__main__.OutputSources'>,)
    attr: {
        '__module__': '__main__',
        '__metaclass__': <class '__main__.MetaPoop'>,
        'PARAM_FILE': '.\\.\\poop.json',
        'POOPY': 'poopy.json',
        'POOP': 'poop.json',
        '__init__': <function __init__ at 0x0000000002384EB8>}
    ------------------------
    cls is <class '__main__.PoopSources'>
    __dict__ is {
        '__module__': '__main__',
        '__metaclass__': <class '__main__.MetaPoop'>,
        'POOPY': 'poopy.json',
        '__init__': <function __init__ at 0x0000000002384EB8>,
        'CRAP': 'crap.json',
        'POOP': 'poop.json',
        'PARAM_FILE': '.\\.\\poop.json',
        '__doc__': None}
    __class__ is <class '__main__.MetaPoop'>
    __class__.__class__ is <type 'type'>
    __class__.__bases__ are (<type 'type'>,)
    ----------------------
    cls is <__main__.PoopSources object at 0x0000000002382978>
    __dict__ is {
        'parameters': {
            u'poop': u'crap',
            u'foo': [1, 2, 3],
            u'bar': {u'this': u'that'}},
        'param_file': '.\\.\\poop.json'}
    __class__ is <class '__main__.PoopSources'>
    __class__.__class__ is <class '__main__.MetaPoop'>
    __class__.__bases__ are (<class '__main__.OutputSources'>,
                             <class __main__.St00pid at 0x0000000002364B28>)
"""


class St00pid(): pass

        
class MetaPoop(type):
    def __new__(cls, name, bases, attr):
        # if we add a subclass here, you may encounter problems:
        # This can happen if your new base has another metaclass other than this
        # one or `type`
        # TypeError: Error when calling the metaclass bases
        #   metaclass conflict: the metaclass of a derived class must be a
        #   (non-strict) subclass of the metaclasses of all its bases
        # And this can happen if you new base has another base that has a
        # conflicting constructor
        # TypeError: Error when calling the metaclass bases
        #   Cannot create a consistent method resolution order (MRO) for bases
        #   object, <conflicting base object>
        if St00pid not in bases:
            bases += (St00pid, )  # be careful
        # changes to name, base and attr can only be made in __new__ before the
        # class is instantiated
        if 'POOP' in attr:
            attr['POOPY'] = 'poopy.json'
        print 'in MetaPoop __new__ method'
        print_cls_name_bases_attr(cls, name, bases, attr)
        return super(MetaPoop, cls).__new__(cls, name, bases, attr)
    def __init__(cls, name, bases, attr):
        super(MetaPoop,cls).__init__(name, bases, attr)
        # make changes to the class directly in __init__ using dot notation.
        # we can add new attributes, but not subclass new bases or change the
        # class name
        if 'POOPY' in attr:
            cls.CRAP = 'crap.json'
        print 'in MetaPoop __init__ method'
        print_cls_name_bases_attr(cls, name, bases, attr)


class PoopSources(OutputSources):
    __metaclass__ = MetaPoop  # 
    POOP = 'poop.json'
    PARAM_FILE = os.path.join(_OUTPUTS, POOP)
    def __init__(self):
        super(PoopSources, self).__init__(PoopSources.PARAM_FILE)

def print_class_dict_class_bases(cls):
    print 'cls is %s' % cls
    print '__dict__ is %s' % cls.__dict__
    print '__class__ is %s' % cls.__class__
    print '__class__.__class__ is %s' % cls.__class__.__class__
    print '__class__.__bases__ are %s' % repr(cls.__class__.__bases__)
    print '----------------------'

if __name__ == "__main__":
    print_class_dict_class_bases(PoopSources)
    po = PoopSources()
    print_class_dict_class_bases(po)
    
    print_class_dict_class_bases(MetaOutputSource)
    # when calling metaclass directly (vs using `__metaclass__` attribute, we
    # provide all of the arguments that `type()` expects.
    PoopOut = MetaOutputSource('PoopOut', (), {})
    print_class_dict_class_bases(PoopOut)
    p = PoopOut('poop.json', {'cool_Fx':100})
    print_class_dict_class_bases(p)
    
    print_class_dict_class_bases(OutputSources)
    o = OutputSources('poop.json')
    print_class_dict_class_bases(o)

poop.json

{
    "poop": "crap",
    "foo": [1, 2, 3],
    "bar": {
        "this": "that"
    }
}

I hope you got an A+ for making a meta-poop primer.