Skip to content

Instantly share code, notes, and snippets.

@chaityacshah

chaityacshah/custom.css

Forked from ChrisBeaumont/custom.css
Created August 25, 2018 02:33
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save chaityacshah/291602f83a9ad25b139bbd44f03bd0a4 to your computer and use it in GitHub Desktop.
Save chaityacshah/291602f83a9ad25b139bbd44f03bd0a4 to your computer and use it in GitHub Desktop.
Download ZIP
Demystifying Python Descriptors
Raw
custom.css
<style>
@font-face {
font-family: "Computer Modern";
src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}
div.cell{
width:800px;
margin-left:16% !important;
margin-right:auto;
}
h1 {
font-family: Helvetica, serif;
}
h4{
margin-top:12px;
margin-bottom: 3px;
}
div.text_cell_render{
font-family: Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
line-height: 145%;
font-size: 130%;
width:800px;
margin-left:auto;
margin-right:auto;
}
div.text_cell_render li {
line-height: 145%;
}
div.text_cell_render code {
color: rgb(40, 114, 43);
font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
font-size: 80%;
}
.CodeMirror{
font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
}
.prompt{
display: None;
}
.text_cell_render h5 {
font-weight: 300;
font-size: 16pt;
color: #4057A1;
font-style: italic;
margin-bottom: .5em;
margin-top: 0.5em;
display: block;
}
.warning{
color: rgb( 240, 20, 20 )
}
</style>
<script>
MathJax.Hub.Config({
TeX: {
extensions: ["AMSmath.js"]
},
tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ]
},
displayAlign: 'center', // Change this to 'center' to center equations.
"HTML-CSS": {
styles: {'.MathJax_Display': {"margin": 4}}
}
});
</script>
Display the source blob
Display the rendered blob
Raw
descriptor_writeup.ipynb
{
"metadata": {
"name": "",
"signature": "sha256:2611929c8797063c4cbdcf40981151339b79f84ad9be28a0c284cffa4a7840eb"
},
"nbformat": 3,
"nbformat_minor": 0,
"worksheets": [
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Python Descriptors Demystified\n",
"\n",
"By [Chris Beaumont](http://chrisbeaumont.org)\n",
"\n",
"Python includes many built-in language features to enable concise, easily-understood code. Some of these niceties include list/set/dictionary comprehensions, properties, and decorators. For the most part, these \"intermediate-level\" language features are well-documented, and easy to learn.\n",
"\n",
"There is one notable exception to this: **descriptors**. For me at least, descriptors were the feature of the core Python language that remained mysterious for the longest time. There are a few reasons for this:\n",
"\n",
" 1. The [official documentation on descriptors](http://docs.python.org/2/howto/descriptor.html) is rather esoteric, and doesn't include good use cases for *why* you might write descriptors (My apologies to Raymond Hettinger, whose *other* Python articles and videos I have found very helpful).\n",
" \n",
" 1. The syntax for writing descriptors is a little weird.\n",
" \n",
" 1. Custom descriptors might be the least-utilized feature of the Python language, so it's hard to find good examples in open source projects.\n",
" \n",
"Nevertheless, descriptors **do** have their use once you figure them out. This document tries to build the argument for what descriptors do, and why you should care.\n",
"\n",
"\n",
"## The punchline: descriptors are reusable properties\n",
"\n",
"Here's what we're building up to: fundamentally, descriptors are properties that you can reuse. That is, descriptors let you write code that looks like this"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"f = Foo()\n",
"b = f.bar\n",
"f.bar = c\n",
"del f.bar"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 14
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and, behind the scenes, calls custom methods when trying to access (`b = f.bar`), assign to (`f.bar = c`), or delete an instance variable (`del f.bar`)\n",
"\n",
"Let's establish why being able to disguise function calls as attribute access is a good thing.\n",
"\n",
"## Properties disguise function calls as attributes\n",
"\n",
"Imagine you are writing some code to organize information about movies (spoiler alert: [these](http://www.imdb.com) [projects](http://www.rottentomatoes.com) beat you to it). You might end up with a movie class that looks like this:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class Movie(object):\n",
" def __init__(self, title, rating, runtime, budget, gross):\n",
" self.title = title\n",
" self.rating = rating\n",
" self.runtime = runtime\n",
" self.budget = budget\n",
" self.gross = gross\n",
" \n",
" def profit(self):\n",
" return self.gross - self.budget"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 15
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You start using this class in other parts of your project, but then you realize something: by mistake, you sometimes assign negative budgets to movies. You decide this is bad, and want the Movie class to forbid this. The first thing you think to try is this:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class Movie(object):\n",
" def __init__(self, title, rating, runtime, budget, gross):\n",
" self.title = title\n",
" self.rating = rating\n",
" self.runtime = runtime\n",
" self.gross = gross\n",
" if budget < 0:\n",
" raise ValueError(\"Negative value not allowed: %s\" % budget)\n",
" self.budget = budget\n",
" \n",
" def profit(self):\n",
" return self.gross - self.budget"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 17
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"But that won't work, because other parts of your code assign values to `Movie.budget` directly -- this new class catches data entry errors within the `__init__` method, but not the cases where somebody tries to run `m.budget = -100` on a pre-existing instance. What's a cinephile pythonista to do?\n",
"\n",
"Luckily, Python **properties** solve this problem. If you've never seen properties before, here's how they work:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class Movie(object):\n",
" def __init__(self, title, rating, runtime, budget, gross):\n",
" self._budget = None\n",
"\n",
" self.title = title\n",
" self.rating = rating\n",
" self.runtime = runtime\n",
" self.gross = gross\n",
" self.budget = budget\n",
" \n",
" @property\n",
" def budget(self):\n",
" return self._budget\n",
" \n",
" @budget.setter\n",
" def budget(self, value):\n",
" if value < 0:\n",
" raise ValueError(\"Negative value not allowed: %s\" % value)\n",
" self._budget = value\n",
" \n",
" def profit(self):\n",
" return self.gross - self.budget\n",
"\n",
" \n",
"m = Movie('Casablanca', 97, 102, 964000, 1300000)\n",
"print m.budget # calls m.budget(), returns result\n",
"try:\n",
" m.budget = -100 # calls budget.setter(-100), and raises ValueError\n",
"except ValueError:\n",
" print \"Woops. Not allowed\""
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"964000\n",
"Woops. Not allowed\n"
]
}
],
"prompt_number": 18
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We specify a getter method with a `@property` decorator, and a setter method with a `@budget.setter` decorator. When we do that, Python automatically calls the getter whenever anybody tries to access the budget. Likewise Python automatically calls `budget.setter` whenever it encounters code like `m.budget = value`.\n",
"\n",
"Take a moment to appreciate how nice it is that Python does this: if properties didn't exist, we'd have to hide all of our instance attributes, and provide lots of explicit methods like `get_budget` and `set_budget`. Code that uses our classes would constantly be calling these getter/setter methods, and would start to look like crufty Java code. Even worse, if we ignored this coding style and just gave direct access to an instance attribute like `budget`, there would be no clean way to later add the non-negativity check -- we would have to retroactively create the `set_budget` method, and search our entire project to change lines like `m.budget = value` to `m.set_budget(value)`. **Gross.**\n",
"\n",
"So properties let you attach custom code to variable getting/setting, while maintaining a simple attribute-like interface for your classes. **Nice.**\n",
"\n",
"## Properties Get Tedious\n",
"\n",
"The main downside to properties is that they aren't reusable. For example, let's assume you want to add the non-negativity check to the `rating`, `runtime`, and `gross` fields as well. Here's the new class"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class Movie(object):\n",
" def __init__(self, title, rating, runtime, budget, gross):\n",
" self._rating = None\n",
" self._runtime = None\n",
" self._budget = None\n",
" self._gross = None\n",
"\n",
" self.title = title\n",
" self.rating = rating\n",
" self.runtime = runtime\n",
" self.gross = gross\n",
" self.budget = budget\n",
" \n",
" #nice\n",
" @property\n",
" def budget(self):\n",
" return self._budget\n",
" \n",
" @budget.setter\n",
" def budget(self, value):\n",
" if value < 0:\n",
" raise ValueError(\"Negative value not allowed: %s\" % value)\n",
" self._budget = value\n",
" \n",
" #ok \n",
" @property\n",
" def rating(self):\n",
" return self._rating\n",
" \n",
" @rating.setter\n",
" def rating(self, value):\n",
" if value < 0:\n",
" raise ValueError(\"Negative value not allowed: %s\" % value)\n",
" self._rating = value\n",
" \n",
" #uhh...\n",
" @property\n",
" def runtime(self):\n",
" return self._runtime\n",
" \n",
" @runtime.setter\n",
" def runtime(self, value):\n",
" if value < 0:\n",
" raise ValueError(\"Negative value not allowed: %s\" % value)\n",
" self._runtime = value \n",
" \n",
" #is this forever?\n",
" @property\n",
" def gross(self):\n",
" return self._gross\n",
" \n",
" @gross.setter\n",
" def gross(self, value):\n",
" if value < 0:\n",
" raise ValueError(\"Negative value not allowed: %s\" % value)\n",
" self._gross = value \n",
" \n",
" def profit(self):\n",
" return self.gross - self.budget"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 19
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"That's **a lot** of code, and a lot of duplicated logic. While properties make the outsides of classes look nice, they don't make the insides of classes look nice.\n",
"\n",
"## Descriptors (Finally)\n",
"\n",
"**This** is the problem that descriptors solve. Descriptors generalize properties, and let you write separate classes for reusable property logic. Here's an example of how they work (for the moment, don't worry about what's inside `NonNegative`):"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from weakref import WeakKeyDictionary\n",
"\n",
"class NonNegative(object):\n",
" \"\"\"A descriptor that forbids negative values\"\"\"\n",
" def __init__(self, default):\n",
" self.default = default\n",
" self.data = WeakKeyDictionary()\n",
" \n",
" def __get__(self, instance, owner):\n",
" # we get here when someone calls x.d, and d is a NonNegative instance\n",
" # instance = x\n",
" # owner = type(x)\n",
" return self.data.get(instance, self.default)\n",
" \n",
" def __set__(self, instance, value):\n",
" # we get here when someone calls x.d = val, and d is a NonNegative instance\n",
" # instance = x\n",
" # value = val\n",
" if value < 0:\n",
" raise ValueError(\"Negative value not allowed: %s\" % value)\n",
" self.data[instance] = value\n",
"\n",
" \n",
"class Movie(object):\n",
" \n",
" #always put descriptors at the class-level\n",
" rating = NonNegative(0)\n",
" runtime = NonNegative(0)\n",
" budget = NonNegative(0)\n",
" gross = NonNegative(0)\n",
" \n",
" def __init__(self, title, rating, runtime, budget, gross):\n",
" self.title = title\n",
" self.rating = rating\n",
" self.runtime = runtime\n",
" self.budget = budget\n",
" self.gross = gross\n",
" \n",
" def profit(self):\n",
" return self.gross - self.budget\n",
" \n",
" \n",
"m = Movie('Casablanca', 97, 102, 964000, 1300000)\n",
"print m.budget # calls Movie.budget.__get__(m, Movie)\n",
"m.rating = 100 # calls Movie.budget.__set__(m, 100)\n",
"try:\n",
" m.rating = -1 # calls Movie.budget.__set__(m, -100)\n",
"except ValueError:\n",
" print \"Woops, negative value\""
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"964000\n",
"Woops, negative value\n"
]
}
],
"prompt_number": 2
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There's some new syntax in here, so let's look at things piece by piece:\n",
"\n",
"`NonNegative` is a descriptor object. It's a descriptor because it defines the `__get__`, `__set__`, or `__delete__` method.\n",
"\n",
"The `Movie` class looks *very* clean. We create 4 descriptors at the class level, and treat them like normal (instance-level) attributes everywhere else. And apparently, the desciptors are checking for non-negative values for us.\n",
" \n",
"#### Accessing a descriptor\n",
"\n",
"When Python sees the line ``print m.budget``, it recognizes that budget is a descriptor with a `__get__` method. Instead of passing `m.budget` to print directly, it calls `Movie.budget.__get__`, and feeds the result of *that* to print. This is similar to what happens when you access a property -- Python automatically calls a method, and returns the result.\n",
"\n",
"`__get__` receives two arguments: the instance object to the left of the period (that is, the `m` object in `m.budget`), and the type of that instance (`Movie`). In some Python [documentation](http://docs.python.org/2/reference/datamodel.html#implementing-descriptors), `Movie` is called the **owner** of the descriptor. If we had asked for `Movie.budget`, Python whould have called `Movie.budget.__get__(None, Movie)`; that is, the fist argument is either an instance of the owner, or ```None```. These input arguments may seem weird to you, but they're there to give you information about what object the descriptor is part of. This will make sense once we look inside the `NonNegative` class.\n",
"\n",
"#### Assigning to a descriptor\n",
"\n",
"When Python sees `m.rating = 100`, Python recognizes `rating` is a descriptor with a ``__set__`` method, and it calls `Movie.rating.__set__(m, 100)`. Like `__get__`, the first argument of `__set__` is the instance to the left of the period (the `m` in `m.rating = 100`). The second argument is the value to the right of the equals sign (100). \n",
"\n",
"#### Deleting a descriptor\n",
"\n",
"For the sake of completeness, if you call `del m.budget`, Python will call `Movie.budget.__delete__(m)`. \n",
"\n",
"#### How NonNegative works\n",
"With this in mind, we can now look to see how the `NonNegative` class works. Each instance of `NonNegative` maintains a dictionary that maps owner instances to data values. When we call `m.budget`, the `__get__` method looks up the data associated with `m`, and returns the result (or a default value, if no such value exists). `__set__` uses the same approach, but includes the extra non-negativity check. We use a `WeakKeyDictionary` instead of a normal `dict` to prevent a memory leak -- we don't want an instance to stay alive simply because it's in the descriptor dictionary, and otherwise unused.\n",
"\n",
"Working with descriptors is slightly awkward. Because they live at the class level, every instance shares the same descriptor. This means that descriptors have to manually manage different states for different object instances, and need to explicitly be passed instances as the first argument of the `__get__`, ``__set__``, and ``__delete__`` methods. \n",
"\n",
"Hopefully, however, this example gives you an idea of what descriptors can be useful for -- they provide a way to organize property logic into isolated classes. If you find yourself repeating the same logic across several properties, that should be a clue to consider whether refactoring that code into a descriptor is worthwhile."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Recipes and Gotchas\n",
"\n",
"\n",
"### Put descriptors at the class level\n",
"\n",
"For descriptors to work properly, they **must** be defined at the class level. If you don't, Python doesn't automatically invoke the `__get__` and `__set__` methods for you:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class Broken(object):\n",
" y = NonNegative(5)\n",
" def __init__(self):\n",
" self.x = NonNegative(0) # NOT a good descriptor\n",
" \n",
"b = Broken()\n",
"print \"X is %s, Y is %s\" % (b.x, b.y)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"X is <__main__.NonNegative object at 0x10432c250>, Y is 5\n"
]
}
],
"prompt_number": 4
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As you can see, accessing the class-level descriptor y automatically calls `__get__`. However, accessing the instance-level descriptor `x` returns the descriptor itself, sans magic.\n",
"\n",
"\n",
"### Make sure to keep instance-level data instance-specific\n",
"\n",
"You might be tempted to write the `NonNegative` descriptor like this"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class BrokenNonNegative(object):\n",
" def __init__(self, default):\n",
" self.value = default\n",
" \n",
" def __get__(self, instance, owner):\n",
" return self.value\n",
" \n",
" def __set__(self, instance, value):\n",
" if value < 0:\n",
" raise ValueError(\"Negative value not allowed: %s\" % value)\n",
" self.value = value\n",
" \n",
"class Foo(object):\n",
" bar = BrokenNonNegative(5) \n",
" \n",
"f = Foo()\n",
"try:\n",
" f.bar = -1\n",
"except ValueError:\n",
" print \"Caught the invalid assignment\""
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Caught the invalid assignment\n"
]
}
],
"prompt_number": 15
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"That seems to work fine. The problem here is that all instances of `Foo` share the same `bar` instance, leading to this flavor of sadness:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class Foo(object):\n",
" bar = BrokenNonNegative(5) \n",
" \n",
"f = Foo()\n",
"g = Foo()\n",
"\n",
"print \"f.bar is %s\\ng.bar is %s\" % (f.bar, g.bar)\n",
"print \"Setting f.bar to 10\"\n",
"f.bar = 10\n",
"print \"f.bar is %s\\ng.bar is %s\" % (f.bar, g.bar) #ouch\n"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"f.bar is 5\n",
"g.bar is 5\n",
"Setting f.bar to 10\n",
"f.bar is 10\n",
"g.bar is 10\n"
]
}
],
"prompt_number": 16
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is why we used the data dictionary in `NonNegative`. The first argument to `__get__` and `__set__` tell us which instance to consider. `NonNegative` uses this argument as a dictionary key, to keep data for each `Foo` instance separate."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class Foo(object):\n",
" bar = NonNegative(5)\n",
" \n",
"f = Foo()\n",
"g = Foo()\n",
"print \"f.bar is %s\\ng.bar is %s\" % (f.bar, g.bar)\n",
"print \"Setting f.bar to 10\"\n",
"f.bar = 10\n",
"print \"f.bar is %s\\ng.bar is %s\" % (f.bar, g.bar) #better"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"f.bar is 5\n",
"g.bar is 5\n",
"Setting f.bar to 10\n",
"f.bar is 10\n",
"g.bar is 5\n"
]
}
],
"prompt_number": 9
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is the most awkward aspect of descriptors (full disclosure: I don't actually understand why Python doesn't let you define descriptors at the instance level, and always dispatch to `__get__` and `__set__`. There must be some reason why this doesn't work. UPDATE: Thanks to [Louie Dinh](https://twitter.com/louiedinh) who pointed me to the reason why: see [this post](https://mail.python.org/pipermail/python-list/2012-January/618565.html) if you're interested).\n",
"\n",
"### Beware unhashable descriptor owners\n",
"\n",
"`NonNegative` uses a dictionary to keep instance-specific data separate. This normally works fine, unless you want to use descriptors with unhashable objects:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class MoProblems(list): #you can't use lists as dictionary keys\n",
" x = NonNegative(5)\n",
" \n",
"m = MoProblems()\n",
"print m.x # womp womp"
],
"language": "python",
"metadata": {},
"outputs": [
{
"ename": "TypeError",
"evalue": "unhashable type: 'MoProblems'",
"output_type": "pyerr",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-8-dd73b177bd8d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mm\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mMoProblems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0;32mprint\u001b[0m \u001b[0mm\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mx\u001b[0m \u001b[0;31m# womp womp\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m<ipython-input-3-6671804ce5d5>\u001b[0m in \u001b[0;36m__get__\u001b[0;34m(self, instance, owner)\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0;31m# instance = x\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 10\u001b[0m \u001b[0;31m# owner = type(x)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minstance\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdefault\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 12\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 13\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m__set__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minstance\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mTypeError\u001b[0m: unhashable type: 'MoProblems'"
]
}
],
"prompt_number": 8
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Because instances of `MoProblems` (which is a subclass of `list`) aren't hashable, they can't be used as keys in the data dictionary for `MoProblems.x`. There are a few ways around this, though none are perfect. The best approach is probably to \"label\" your descriptors"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class Descriptor(object):\n",
" \n",
" def __init__(self, label):\n",
" self.label = label\n",
" \n",
" def __get__(self, instance, owner):\n",
" print '__get__', instance, owner\n",
" return instance.__dict__.get(self.label)\n",
" \n",
" def __set__(self, instance, value):\n",
" print '__set__'\n",
" instance.__dict__[self.label] = value\n",
" \n",
"\n",
"class Foo(list):\n",
" x = Descriptor('x')\n",
" y = Descriptor('y')\n",
" \n",
"f = Foo()\n",
"f.x = 5\n",
"print f.x"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"__set__\n",
"__get__ [] <class '__main__.Foo'>\n",
"5\n"
]
}
],
"prompt_number": 2
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This relies on a highly non-obvious detail of Python's [method resolution order](http://python-history.blogspot.com/2010/06/method-resolution-order.html). We label each descriptor in Foo with the same name as the variable that we assign the descriptor to (for example, ``x = Descriptor('x')``). The descriptor then stores instance-specific data in ``f.__dict__['x']``. This dictionary entry would normally be what Python returns when we ask for `f.x`. However, because `Foo.x` is a descriptor, Python doesn't use `f.__dict__['x']` normally, and the descriptor can safely store stuff there. Just make sure you don't label the descriptor anything else:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class Foo(object):\n",
" x = Descriptor('y')\n",
" \n",
"f = Foo()\n",
"f.x = 5\n",
"print f.x\n",
"\n",
"f.y = 4 #oh no!\n",
"print f.x"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"__set__\n",
"__get__ <__main__.Foo object at 0x10432c810> <class '__main__.Foo'>\n",
"5\n",
"__get__ <__main__.Foo object at 0x10432c810> <class '__main__.Foo'>\n",
"4\n"
]
}
],
"prompt_number": 10
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"I don't love this pattern, since it's fragile and subtle, but it's fairly common. And it works for unhashable owner classes. David Beazley uses it in his [books](http://www.amazon.com/Python-Essential-Reference-4th-Edition/dp/0672329786/)\n",
"\n",
"\n",
"### Labeled Descriptors with Metaclasses\n",
"\n",
"Because descriptor labels match the variable name they are assigned to, some people use metaclasses to take care of this bookkeeping automatically:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class Descriptor(object):\n",
" def __init__(self):\n",
" #notice we aren't setting the label here\n",
" self.label = None\n",
" \n",
" def __get__(self, instance, owner):\n",
" print '__get__. Label = %s' % self.label\n",
" return instance.__dict__.get(self.label, None)\n",
" \n",
" def __set__(self, instance, value):\n",
" print '__set__'\n",
" instance.__dict__[self.label] = value\n",
"\n",
" \n",
"class DescriptorOwner(type):\n",
" def __new__(cls, name, bases, attrs):\n",
" # find all descriptors, auto-set their labels\n",
" for n, v in attrs.items():\n",
" if isinstance(v, Descriptor):\n",
" v.label = n\n",
" return super(DescriptorOwner, cls).__new__(cls, name, bases, attrs)\n",
"\n",
" \n",
"class Foo(object):\n",
" __metaclass__ = DescriptorOwner\n",
" x = Descriptor()\n",
" \n",
"f = Foo()\n",
"f.x = 10\n",
"print f.x\n",
" "
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"__set__\n",
"__get__. Label = x\n",
"10\n"
]
}
],
"prompt_number": 15
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"I won't explain the details of metaclasses -- David Beazley's tutorial at the bottom of this article covers them. The main point is that the metaclass auto-assigns descriptor labels, to match the variable name that each descriptor is assigned to.\n",
"\n",
"While this solves the problem of mismatched descriptor labels and variable names, it does so by adding all the complexity of metaclasses. You can decide if this is worth the hassle, but I have my doubts.\n",
"\n",
"### Accessing Descriptor Methods\n",
"\n",
"Descriptors are just classes, and you may want to add other methods to them. For example, descriptors are a great way to implement callback properties. Say we want a class to notify us whenever part of its state changes. Here's most of the code to do that"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class CallbackProperty(object):\n",
" \"\"\"A property that will alert observers when upon updates\"\"\"\n",
" def __init__(self, default=None):\n",
" self.data = WeakKeyDictionary()\n",
" self.default = default\n",
" self.callbacks = WeakKeyDictionary()\n",
" \n",
" def __get__(self, instance, owner):\n",
" return self.data.get(instance, self.default)\n",
" \n",
" def __set__(self, instance, value): \n",
" for callback in self.callbacks.get(instance, []):\n",
" # alert callback function of new value\n",
" callback(value)\n",
" self.data[instance] = value\n",
" \n",
" def add_callback(self, instance, callback):\n",
" \"\"\"Add a new function to call everytime the descriptor updates\"\"\"\n",
" #but how do we get here?!?!\n",
" if instance not in self.callbacks:\n",
" self.callbacks[instance] = []\n",
" self.callbacks[instance].append(callback)\n",
" \n",
"class BankAccount(object):\n",
" balance = CallbackProperty(0)\n",
" \n",
"def low_balance_warning(value):\n",
" if value < 100:\n",
" print \"You are poor\"\n",
" \n",
"ba = BankAccount()\n",
"\n",
"# will not work -- try it\n",
"#ba.balance.add_callback(ba, low_balance_warning)"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 3
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is a promising pattern -- we can attach custom callback functions to respond to state changes within a class, without having to modify the class code at all. That's a lovely separation of concerns. All we need to do now is call `ba.balance.add_callback(ba, low_balance_warning)`, so that `low_balance_warning` is called whenever `balance` changes.\n",
"\n",
"But how do we do that? Descriptors *always* call `__get__` when we try to access them. It would seem that the ``add_callback`` method is unreachable! The trick is to take advantage of the special case that, when accessed from the class level, the first argument to `__get__` is `None`:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class CallbackProperty(object):\n",
" \"\"\"A property that will alert observers when upon updates\"\"\"\n",
" def __init__(self, default=None):\n",
" self.data = WeakKeyDictionary()\n",
" self.default = default\n",
" self.callbacks = WeakKeyDictionary()\n",
" \n",
" def __get__(self, instance, owner):\n",
" if instance is None:\n",
" return self \n",
" return self.data.get(instance, self.default)\n",
" \n",
" def __set__(self, instance, value):\n",
" for callback in self.callbacks.get(instance, []):\n",
" # alert callback function of new value\n",
" callback(value)\n",
" self.data[instance] = value\n",
" \n",
" def add_callback(self, instance, callback):\n",
" \"\"\"Add a new function to call everytime the descriptor within instance updates\"\"\"\n",
" if instance not in self.callbacks:\n",
" self.callbacks[instance] = []\n",
" self.callbacks[instance].append(callback)\n",
" \n",
"class BankAccount(object):\n",
" balance = CallbackProperty(0)\n",
" \n",
"def low_balance_warning(value):\n",
" if value < 100:\n",
" print \"You are now poor\"\n",
" \n",
"ba = BankAccount()\n",
"BankAccount.balance.add_callback(ba, low_balance_warning)\n",
"\n",
"ba.balance = 5000\n",
"print \"Balance is %s\" % ba.balance\n",
"ba.balance = 99\n",
"print \"Balance is %s\" % ba.balance\n"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Balance is 5000\n",
"You are now poor\n",
"Balance is 99\n"
]
}
],
"prompt_number": 5
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Fin\n",
"\n",
"Hopefully, you now have an understanding of what descriptors are, and when they are useful. Go forth and refactor.\n",
"\n",
"\n",
"### Acknowledgements\n",
"\n",
"The CSS on this page is adapted from Cam Davidson-Pilon's awesome and gorgeous [book](http://nbviewer.ipython.org/urls/raw.github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/master/Chapter1_Introduction/Chapter1_Introduction.ipynb).\n",
"\n",
"There were some relevant talks and tutorials about descriptors and properties at PyCon 2013:\n",
"\n",
" * [Encapsulation With Descriptors](http://pyvideo.org/video/1760/encapsulation-with-descriptors) by Luciano Ramalho\n",
" * [Python 3 Metaprogramming](http://pyvideo.org/video/1716/python-3-metaprogramming) by David Beazley\n",
" * [Python's class development toolkit](http://pyvideo.org/video/1779/pythons-class-development-toolkit) by Raymond Hettinger"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"#This makes everything pretty\n",
"\n",
"from IPython.core.display import HTML\n",
"from urllib import urlopen\n",
"def css_styling():\n",
" styles = open('custom.css', 'r').read()\n",
" return HTML(styles)\n",
"css_styling()"
],
"language": "python",
"metadata": {},
"outputs": [
{
"html": [
"<style>\n",
" @font-face {\n",
" font-family: \"Computer Modern\";\n",
" src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');\n",
" }\n",
" div.cell{\n",
" width:800px;\n",
" margin-left:16% !important;\n",
" margin-right:auto;\n",
" }\n",
" h1 {\n",
" font-family: Helvetica, serif;\n",
" }\n",
" h4{\n",
" margin-top:12px;\n",
" margin-bottom: 3px;\n",
" }\n",
" div.text_cell_render{\n",
" font-family: Computer Modern, \"Helvetica Neue\", Arial, Helvetica, Geneva, sans-serif;\n",
" line-height: 145%;\n",
" font-size: 130%;\n",
" width:800px;\n",
" margin-left:auto;\n",
" margin-right:auto;\n",
" }\n",
"\n",
" div.text_cell_render li {\n",
" line-height: 145%;\n",
" }\n",
"\n",
" div.text_cell_render code {\n",
" color: rgb(40, 114, 43);\n",
" font-family: \"Source Code Pro\", source-code-pro,Consolas, monospace;\n",
" font-size: 80%;\n",
" }\n",
"\n",
" .CodeMirror{\n",
" font-family: \"Source Code Pro\", source-code-pro,Consolas, monospace;\n",
" }\n",
"\n",
" .prompt{\n",
" display: None;\n",
" }\n",
" .text_cell_render h5 {\n",
" font-weight: 300;\n",
" font-size: 16pt;\n",
" color: #4057A1;\n",
" font-style: italic;\n",
" margin-bottom: .5em;\n",
" margin-top: 0.5em;\n",
" display: block;\n",
" }\n",
"\n",
" .warning{\n",
" color: rgb( 240, 20, 20 )\n",
" }\n",
"</style>\n",
"<script>\n",
" MathJax.Hub.Config({\n",
" TeX: {\n",
" extensions: [\"AMSmath.js\"]\n",
" },\n",
" tex2jax: {\n",
" inlineMath: [ ['$','$'], [\"\\\\(\",\"\\\\)\"] ],\n",
" displayMath: [ ['$$','$$'], [\"\\\\[\",\"\\\\]\"] ]\n",
" },\n",
" displayAlign: 'center', // Change this to 'center' to center equations.\n",
" \"HTML-CSS\": {\n",
" styles: {'.MathJax_Display': {\"margin\": 4}}\n",
" }\n",
" });\n",
"</script>"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 1,
"text": [
"<IPython.core.display.HTML at 0x1022c1cd0>"
]
}
],
"prompt_number": 1
},
{
"cell_type": "code",
"collapsed": false,
"input": [],
"language": "python",
"metadata": {},
"outputs": []
}
],
"metadata": {}
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment