Skip to content

Instantly share code, notes, and snippets.

@3kwa

3kwa/__slots__.ipynb

Created April 5, 2013 07:35
Show Gist options
  • Save 3kwa/5317328 to your computer and use it in GitHub Desktop.
Save 3kwa/5317328 to your computer and use it in GitHub Desktop.
Download ZIP
SyPy presentation on __slots__ using iPython notebook
Display the source blob
Display the rendered blob
Raw
__slots__.ipynb
{
"metadata": {
"name": "__slots__"
},
"nbformat": 3,
"nbformat_minor": 0,
"worksheets": [
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# \\__slots__ flyweight pattern made easy\n",
"\n",
"A flyweight is an object that minimizes memory use by sharing as much data as\n",
"possible with other similar objects; it is a way to use objects in large\n",
"numbers when a simple repeated representation would use an unacceptable amount\n",
"of memory"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"import random\n",
"\n",
"class Pokies(object):\n",
" \"\"\"\n",
" The Aussie one arm bandit\n",
" \"\"\"\n",
"\n",
" def __init__(self, reels, symbols):\n",
" self.reels = reels\n",
" self.symbols = symbols\n",
"\n",
" def draw(self):\n",
" \"\"\"\n",
" Random draw of reels * symbols\n",
" \"\"\"\n",
" return tuple(random.randint(1, self.symbols) for _ in range(self.reels))\n",
" \n",
"pokies = Pokies(3, 10)\n",
"pokies.draw()"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 2,
"text": [
"(4, 5, 2)"
]
}
],
"prompt_number": 2
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"dir(pokies)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 3,
"text": [
"['__class__',\n",
" '__delattr__',\n",
" '__dict__',\n",
" '__doc__',\n",
" '__format__',\n",
" '__getattribute__',\n",
" '__hash__',\n",
" '__init__',\n",
" '__module__',\n",
" '__new__',\n",
" '__reduce__',\n",
" '__reduce_ex__',\n",
" '__repr__',\n",
" '__setattr__',\n",
" '__sizeof__',\n",
" '__str__',\n",
" '__subclasshook__',\n",
" '__weakref__',\n",
" 'draw',\n",
" 'reels',\n",
" 'symbols']"
]
}
],
"prompt_number": 3
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# \\__dict__\n",
"\n",
"The attribute dictionary of an instance"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pokies.__dict__"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 4,
"text": [
"{'reels': 3, 'symbols': 10}"
]
}
],
"prompt_number": 4
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pokies.owner = 'Aristocrat'\n",
"pokies.__dict__"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 5,
"text": [
"{'owner': 'Aristocrat', 'reels': 3, 'symbols': 10}"
]
}
],
"prompt_number": 5
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"del pokies.owner\n",
"pokies.__dict__"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 6,
"text": [
"{'reels': 3, 'symbols': 10}"
]
}
],
"prompt_number": 6
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# \\__weakref__\n",
"\n",
"A weak reference to an object is not enough to keep the object alive: when the only remaining references to a referent are weak references, garbage collection is free to destroy the referent and reuse its memory for something else. A primary use for weak references is to implement caches or mappings holding large objects, where it\u2019s desired that a large object not be kept alive solely because it appears in a cache or mapping."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"import weakref\n",
"ref = weakref.ref(pokies)\n",
"ref"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 7,
"text": [
"<weakref at 0x102daff70; to 'Pokies' at 0x102db5cd0>"
]
}
],
"prompt_number": 7
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"One accesses the referent by calling the weak reference:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"ref().draw()"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 8,
"text": [
"(6, 4, 6)"
]
}
],
"prompt_number": 8
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"pokies.__weakref__"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 9,
"text": [
"<weakref at 0x102daff70; to 'Pokies' at 0x102db5cd0>"
]
}
],
"prompt_number": 9
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# A flyweight minimizes memory use\n",
"\n",
"The __slots__ declaration takes a sequence of instance variables and reserves just enough space in each instance to hold a value for each variable. Space is saved because __dict__ is not created for each instance."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class FlyweightPokies(object):\n",
" \"\"\"\n",
" One arm bandit with smaller memory footprint\n",
" \"\"\"\n",
"\n",
" __slots__ = ['reels', 'symbols']\n",
"\n",
" def __init__(self, reels, symbols):\n",
" self.reels = reels\n",
" self.symbols = symbols\n",
"\n",
" def draw(self):\n",
" \"\"\"\n",
" Random draw of reels x symbols\n",
" \"\"\"\n",
" return tuple(random.randint(1, self.symbols) for _ in range(self.reels))\n",
" \n",
"flyweight = FlyweightPokies(3, 10)\n",
"flyweight.draw()"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 10,
"text": [
"(9, 5, 5)"
]
}
],
"prompt_number": 10
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# What is the difference?"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"set(dir(pokies)) - set(dir(flyweight))"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 11,
"text": [
"set(['__dict__', '__weakref__'])"
]
}
],
"prompt_number": 11
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There is no \\__dict__ so no way to add attribute to the instance:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"flyweight.owner = 'Aristocrat'"
],
"language": "python",
"metadata": {},
"outputs": [
{
"ename": "AttributeError",
"evalue": "'FlyweightPokies' object has no attribute 'owner'",
"output_type": "pyerr",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-12-a600c6f9aae6>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mflyweight\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mowner\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'Aristocrat'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;31mAttributeError\u001b[0m: 'FlyweightPokies' object has no attribute 'owner'"
]
}
],
"prompt_number": 12
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There is no \\__weakref__ so no way to create a weak reference to the instance:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"weakref.ref(flyweight)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"ename": "TypeError",
"evalue": "cannot create weak reference to 'FlyweightPokies' object",
"output_type": "pyerr",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-13-e09859bf60e5>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mweakref\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mref\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mflyweight\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;31mTypeError\u001b[0m: cannot create weak reference to 'FlyweightPokies' object"
]
}
],
"prompt_number": 13
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We could make a *weakreferencable* pokies class though, by adding \\__weakref__ to list of attributes in \\__slots__:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class WeakPokies(object):\n",
" \"\"\"\n",
" A weakrefencable one arm bandit with smallish memory footprint \n",
" \"\"\"\n",
"\n",
" __slots__ = ['reels', 'symbols', '__weakref__']\n",
"\n",
" def __init__(self, reels, symbols):\n",
" self.reels = reels\n",
" self.symbols = symbols\n",
"\n",
" def draw(self):\n",
" \"\"\"\n",
" Random draw of reels x symbols\n",
" \"\"\"\n",
" return tuple(random.randint(1, self.symbols) for _ in range(self.reels))\n",
" \n",
"weak = WeakPokies(3, 10)\n",
"weak.draw()"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 20,
"text": [
"(10, 10, 7)"
]
}
],
"prompt_number": 20
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"weakref.ref(weak)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 21,
"text": [
"<weakref at 0x103864158; to 'WeakPokies' at 0x10385f7a0>"
]
}
],
"prompt_number": 21
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# ~25% memory usage\n",
"\n",
"The dramatic change is the memory footprint of the flyweight instance compare to a run of the mill object:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"%load_ext memory_profiler"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 15
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"%memit [Pokies(3, 10) for _ in xrange(int(1e6))]"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"maximum of 1: 360.085938 MB per loop\n"
]
}
],
"prompt_number": 16
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"%memit [FlyweightPokies(3, 10) for _ in xrange(int(1e6))]"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"maximum of 1: 91.593750 MB per loop\n"
]
}
],
"prompt_number": 17
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"%memit [WeakPokies(3, 10) for _ in xrange(int(1e6))]"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"maximum of 1: 97.730469 MB per loop\n"
]
}
],
"prompt_number": 22
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Do it last, if ever\n",
"\n",
"\\__slots__ is an optimization that is only relevant if you are going to create **MANY** instances of an object and which you should do last if ever!"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class BrokenPokies(object):\n",
" \"\"\"\n",
" Getting it wrong raises an exception on instantiation\n",
" \"\"\"\n",
"\n",
" __slots__ = ()\n",
"\n",
" def __init__(self, reels, symbols):\n",
" self.reels = reels\n",
" self.symbols = symbols\n",
" \n",
"broken = BrokenPokies(3, 10)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"ename": "AttributeError",
"evalue": "'BrokenPokies' object has no attribute 'reels'",
"output_type": "pyerr",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-23-564a4def7260>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 10\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msymbols\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msymbols\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 11\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 12\u001b[0;31m \u001b[0mbroken\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mBrokenPokies\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m<ipython-input-23-564a4def7260>\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, reels, symbols)\u001b[0m\n\u001b[1;32m 7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m__init__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreels\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msymbols\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 9\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreels\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mreels\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 10\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msymbols\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msymbols\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 11\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mAttributeError\u001b[0m: 'BrokenPokies' object has no attribute 'reels'"
]
}
],
"prompt_number": 23
},
{
"cell_type": "code",
"collapsed": false,
"input": [],
"language": "python",
"metadata": {},
"outputs": []
}
],
"metadata": {}
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment