Skip to content

Instantly share code, notes, and snippets.

@DRMacIver

DRMacIver/Python Inconsistencies.ipynb

Last active October 26, 2022 11:50
Show Gist options
  • Star 15 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save DRMacIver/47519854a0e62538b542 to your computer and use it in GitHub Desktop.
Save DRMacIver/47519854a0e62538b542 to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
Python Inconsistencies.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Python Inconsistencies and Eccentricities\n",
"\n",
"This is a collection of eccentricities and inconsistencies I have run into in Python, mostly in the course of writing [Hypothesis](https://hypothesis.readthedocs.org/en/latest/).\n",
"\n",
"Most of them aren't terrible and are merely weird or annoying, and most languages have warts like this. Python feels like it has an atypically high rate of them but maybe I've just looked a lot more closely.\n",
"\n",
"All of these examples are Python 2, with notes where they no longer apply in Python 3. Python 3 has a few new eccentricities of its own but is mostly a bit better in this regard.\n",
"\n",
"These are in no particular order except the one I remembered them in."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Containment and equality"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"x = float('nan')\n",
"x == x"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So far so \"good\". This is the normal IEEE behaviour for floats."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"float('nan') in [float('nan')]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Expected behaviour still. There is no value in the list equal to float('nan')!"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"x in [x]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The defined behaviour of contains for lists and tuples is that x in y if any(x is s or x == s for s in y). This is [documented in Python 3](https://docs.python.org/3.5/reference/expressions.html#not-in) but is true (and [contrary to the documentation](https://docs.python.org/2/reference/expressions.html#not-in) in Python 2."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that you will get slightly different behaviour in pypy because float('nan') is float('nan') is always True in pypy. Also there is a bug in some versions where x in [x] will return False but x in [x, \"hi\"] will return True."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## When x += y is different from x = x + y"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"x = [1]\n",
"y = x\n",
"x += [2]"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[1, 2]"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"x"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[1, 2]"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"x = (1,)\n",
"y = x\n",
"x += (2,)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(1, 2)"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"x"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(1,)"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"+= is supported for everything that has + defined, but you can override \\_\\_iadd\\_\\_ in which case x += y is equivalent to x = x.\\_\\_iadd\\_\\_(y). For lists, iadd mutates x in place."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is particularly fun when you put mutable types inside immutable ones:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"x = ([],)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false
},
"outputs": [
{
"ename": "TypeError",
"evalue": "'tuple' object does not support item assignment",
"output_type": "error",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[1;31mTypeError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m<ipython-input-11-bbb20d4e3867>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mx\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m+=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[1;31mTypeError\u001b[0m: 'tuple' object does not support item assignment"
]
}
],
"source": [
"x[0] += [1]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As you'd hope, we got an error on item assignment to a tuple! But..."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[1]"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"x[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Because the assignment mutated it in place, even though we were not able to update the value in the index it \"worked\" anyway."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## type(x + 1 - 1) != type(x)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"9223372036854775807"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import sys\n",
"sys.maxint"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"int"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"type(sys.maxint)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"9223372036854775807L"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"(sys.maxint + 1) - 1"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"long"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"type((sys.maxint + 1) - 1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This one is of course gone in Python 3 because the int/long distinction is gone."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Non-obvious associativity"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"0 == 1 is False"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"(0 == 1) is False"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"0 == (1 is False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The above is part of operator chaining, It's interpreted as (0 == 1) and (1 is False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## How do you look up the method resolution order?"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[object]"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"object.mro()"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[str, basestring, object]"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"str.mro()"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"collapsed": false
},
"outputs": [
{
"ename": "TypeError",
"evalue": "descriptor 'mro' of 'type' object needs an argument",
"output_type": "error",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[1;31mTypeError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m<ipython-input-22-fd092325fdfa>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mtype\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmro\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[1;31mTypeError\u001b[0m: descriptor 'mro' of 'type' object needs an argument"
]
}
],
"source": [
"type.mro()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Oh right. This could be a type. So clearly what we should have done is..."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[type, object]"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"type.mro(type)"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[object]"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"type.mro(object)"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[str, basestring, object]"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"type.mro(str)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Looks good, right?"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"'Hi'"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"class Foo(object):\n",
" def foo(self):\n",
" return \"Hi\"\n",
"\n",
"class AddsFoo(type):\n",
" def mro(self):\n",
" return super(AddsFoo, self).mro() + [Foo]\n",
" \n",
"class Bar(object):\n",
" __metaclass__ = AddsFoo\n",
" \n",
"Bar().foo()"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[__main__.Bar, object, __main__.Foo]"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"Bar.mro()"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[__main__.Bar, object]"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"type.mro(Bar)"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[__main__.Bar, object, __main__.Foo]"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"type(Bar).mro(Bar)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is incorrect to call type.mro(x) because x might have a custom metaclass. The only correct way to look up the mro of an arbitrary python type is type(x).mro(x)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## How does repr work?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here's a problem I had recently. In 2.7 \\_\\_repr\\_\\_ must return only ascii strings. It's OK to return a unicode string, but it will be interpreted as ascii."
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"class CustomRepr(object):\n",
" def __init__(self, rep):\n",
" self.rep = rep\n",
"\n",
" def __repr__(self):\n",
" return self.rep "
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"foo"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"CustomRepr(\"foo\")"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"foo"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"CustomRepr(u\"foo\")"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"collapsed": false
},
"outputs": [
{
"ename": "UnicodeEncodeError",
"evalue": "'ascii' codec can't encode character u'\\u2603' in position 0: ordinal not in range(128)",
"output_type": "error",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[1;31mUnicodeEncodeError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m/home/david/.pyenv/versions/2.7.10/lib/python2.7/site-packages/IPython/core/formatters.pyc\u001b[0m in \u001b[0;36m__call__\u001b[1;34m(self, obj)\u001b[0m\n\u001b[0;32m 688\u001b[0m \u001b[0mtype_pprinters\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtype_printers\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 689\u001b[0m deferred_pprinters=self.deferred_printers)\n\u001b[1;32m--> 690\u001b[1;33m \u001b[0mprinter\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpretty\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 691\u001b[0m \u001b[0mprinter\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mflush\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 692\u001b[0m \u001b[1;32mreturn\u001b[0m \u001b[0mstream\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgetvalue\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
"\u001b[1;32m/home/david/.pyenv/versions/2.7.10/lib/python2.7/site-packages/IPython/lib/pretty.pyc\u001b[0m in \u001b[0;36mpretty\u001b[1;34m(self, obj)\u001b[0m\n\u001b[0;32m 407\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0mcallable\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmeth\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 408\u001b[0m \u001b[1;32mreturn\u001b[0m \u001b[0mmeth\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcycle\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 409\u001b[1;33m \u001b[1;32mreturn\u001b[0m \u001b[0m_default_pprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcycle\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 410\u001b[0m \u001b[1;32mfinally\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 411\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mend_group\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
"\u001b[1;32m/home/david/.pyenv/versions/2.7.10/lib/python2.7/site-packages/IPython/lib/pretty.pyc\u001b[0m in \u001b[0;36m_default_pprint\u001b[1;34m(obj, p, cycle)\u001b[0m\n\u001b[0;32m 527\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0m_safe_getattr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mklass\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'__repr__'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mNone\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32min\u001b[0m \u001b[0m_baseclass_reprs\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 528\u001b[0m \u001b[1;31m# A user-provided repr. Find newlines and replace them with p.break_()\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 529\u001b[1;33m \u001b[0m_repr_pprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mp\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcycle\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 530\u001b[0m \u001b[1;32mreturn\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 531\u001b[0m \u001b[0mp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mbegin_group\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'<'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
"\u001b[1;32m/home/david/.pyenv/versions/2.7.10/lib/python2.7/site-packages/IPython/lib/pretty.pyc\u001b[0m in \u001b[0;36m_repr_pprint\u001b[1;34m(obj, p, cycle)\u001b[0m\n\u001b[0;32m 709\u001b[0m \u001b[1;34m\"\"\"A pprint that just redirects to the normal repr function.\"\"\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 710\u001b[0m \u001b[1;31m# Find newlines and replace them with p.break_()\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 711\u001b[1;33m \u001b[0moutput\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mrepr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 712\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0midx\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0moutput_line\u001b[0m \u001b[1;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0moutput\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msplitlines\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 713\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0midx\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
"\u001b[1;31mUnicodeEncodeError\u001b[0m: 'ascii' codec can't encode character u'\\u2603' in position 0: ordinal not in range(128)"
]
}
],
"source": [
"CustomRepr(u\"☃\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is annoyingly common to get this wrong. Suppose I wanted to display an object that gets it wrong? I am perfectly able to display the unicode, but repr() won't let me."
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"☃\n"
]
}
],
"source": [
"print(CustomRepr(\"☃\").__repr__())"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<object object at 0x7fb8c2fe3fa0>\n"
]
}
],
"source": [
"print(object().__repr__())"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {
"collapsed": false
},
"outputs": [
{
"ename": "TypeError",
"evalue": "descriptor '__repr__' of 'object' object needs an argument",
"output_type": "error",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[1;31mTypeError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m<ipython-input-36-07552b3772a0>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[1;32mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mobject\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__repr__\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[1;31mTypeError\u001b[0m: descriptor '__repr__' of 'object' object needs an argument"
]
}
],
"source": [
"print(object.__repr__())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This has the same resolution as the mro problem."
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def safe_repr(x):\n",
" return type(x).__repr__(x)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\"But David!\" you say. \"What if \\_\\_repr\\_\\_ is assigned on the instance rather than the class?"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"<__main__.NoCustomRepr at 0x7fb8b22f3590>"
]
},
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"class NoCustomRepr(object):\n",
" pass\n",
"\n",
"x = NoCustomRepr()\n",
"x.__repr__ = \"Hi\"\n",
"x"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[For new-style classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary.](https://docs.python.org/2/reference/datamodel.html#special-method-lookup-for-new-style-classes)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So far so good.\n",
"\n",
"[Except...](https://bitbucket.org/pypy/pypy/issues/2083/)\n",
"\n",
"This workaround is no longer needed in Python 3, but if it were the behaviour would be the same."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Attributes going missing"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"class HasAProperty(object):\n",
" @property\n",
" def stuff(self):\n",
" return self.oops_does_not_exist\n",
" \n",
"hasattr(HasAProperty, 'stuff')"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"hasattr(HasAProperty(), 'stuff')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"hasattr works by calling the property and seeing if you get an AttributeError. It doesn't care where the AttributeError comes from."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Never override methods of builtin types"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"class BonusList(list):\n",
" def __iter__(self):\n",
" for x in super(BonusList, self).__iter__():\n",
" yield x\n",
" yield 1"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[1]"
]
},
"execution_count": 42,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"BonusList()"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[1]"
]
},
"execution_count": 43,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"list(BonusList())"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"()"
]
},
"execution_count": 44,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tuple(BonusList())"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(1,)"
]
},
"execution_count": 45,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tuple(iter(BonusList()))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is the sort of inconsistency you can get from C extensions using the concrete methods rather than the abstract protocol. Because a lot of Python builtins are implemented in C, you can run into this sort of thing.\n",
"\n",
"Basically the only safe thing to do is never override methods of builtins.\n",
"\n",
"This behaviour is the same on Python 3 but correct on pypy because pypy has an actually sensible implementation."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## How do scopes of class bodies work?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Class bodies act a lot like normal scopes. For example"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Hello world\n"
]
}
],
"source": [
"class Hello(object):\n",
" x = \"world\"\n",
" print(\"Hello %s\" % (x,))"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {
"collapsed": false
},
"outputs": [
{
"ename": "NameError",
"evalue": "global name 'e' is not defined",
"output_type": "error",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[1;31mNameError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m<ipython-input-47-ff19e5b0b9d8>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[1;32mclass\u001b[0m \u001b[0mA\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mobject\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 2\u001b[0m \u001b[0ma\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;36m10\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 3\u001b[0m \u001b[0mb\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mrange\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ma\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 4\u001b[0m \u001b[0mc\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mx\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mb\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 5\u001b[0m \u001b[0md\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mlist\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mx\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mc\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
"\u001b[1;32m<ipython-input-47-ff19e5b0b9d8>\u001b[0m in \u001b[0;36mA\u001b[1;34m()\u001b[0m\n\u001b[0;32m 5\u001b[0m \u001b[0md\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mlist\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mx\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mc\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 6\u001b[0m \u001b[0me\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0md\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ma\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 7\u001b[1;33m \u001b[0mf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mlist\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ma\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[1;32m<ipython-input-47-ff19e5b0b9d8>\u001b[0m in \u001b[0;36m<genexpr>\u001b[1;34m((i,))\u001b[0m\n\u001b[0;32m 5\u001b[0m \u001b[0md\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mlist\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mx\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mc\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 6\u001b[0m \u001b[0me\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0md\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ma\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 7\u001b[1;33m \u001b[0mf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mlist\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ma\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[1;31mNameError\u001b[0m: global name 'e' is not defined"
]
}
],
"source": [
"class A(object):\n",
" a = 10\n",
" b = range(a)\n",
" c = [x for x in b]\n",
" d = list(x for x in c)\n",
" e = [d[i] for i in range(a)]\n",
" f = list(e[i] for i in range(a))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For some reason e was not in scope there even though all the previous ones are. This would have worked fine if we were in a method rather than class body:"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def A():\n",
" a = 10\n",
" b = range(a)\n",
" c = [x for x in b]\n",
" d = list(x for x in c)\n",
" e = [d[i] for i in range(a)]\n",
" f = list(e[i] for i in range(a))\n",
" \n",
"A()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The reason for this is that Python distinguishes between scopes and execution frames. The latter are what are captured by function definitions. [After a class body has finished executing, the execution frame is discarded](https://docs.python.org/2/reference/compound_stmts.html#class-definitions), which is why class level variables are not visible in function definitions. What is in scope in a generator body is complicated: The top level collection being iterated over is received from the scope, but the body of the generator must capture the execution frame."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## When are types of two different values equal?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Python is very lax about equality of different numericish types"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 49,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"{0: set([1])} == {False: frozenset([True])} == {0.0: set([1.0])}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note in particular that a frozenset is equal to a set with the same (or equivalent) contents. This is different from the tuple/list behaviour:"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 50,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"[] == ()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exceptions holding on to values unexpectedly"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {
"collapsed": false
},
"outputs": [
{
"ename": "AssertionError",
"evalue": "",
"output_type": "error",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[1;31mAssertionError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m<ipython-input-51-03f445bff835>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m 33\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 34\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 35\u001b[1;33m \u001b[0mtest_gc\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[1;32m<ipython-input-51-03f445bff835>\u001b[0m in \u001b[0;36mtest_gc\u001b[1;34m()\u001b[0m\n\u001b[0;32m 30\u001b[0m \u001b[1;31m# The Key() argument went out of scope immediately. When we ran the GC\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 31\u001b[0m \u001b[1;31m# it should definitely have had the weakref to it cleared.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 32\u001b[1;33m \u001b[1;32massert\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mcache\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 33\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 34\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
"\u001b[1;31mAssertionError\u001b[0m: "
]
}
],
"source": [
"\n",
"import gc\n",
"from weakref import WeakKeyDictionary\n",
"\n",
"\n",
"class Key(object):\n",
" pass\n",
"\n",
"\n",
"def run(cache, template):\n",
" cache[template] = 1\n",
" # Note: Not raising the ValueError causes this to work. Any exception will\n",
" # cause the same problem though.\n",
" raise ValueError()\n",
"\n",
"\n",
"def test_gc():\n",
" cache = WeakKeyDictionary()\n",
" # Extracting this whole try/except into its own function and passing in\n",
" # cache as an argument causes this to work.\n",
" try:\n",
" # Note: Inlining run here causes this to work\n",
" run(\n",
" cache, Key()\n",
" )\n",
" except ValueError:\n",
" pass\n",
" gc.collect()\n",
"\n",
" # The Key() argument went out of scope immediately. When we ran the GC\n",
" # it should definitely have had the weakref to it cleared.\n",
" assert not cache\n",
"\n",
"\n",
"test_gc()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The reason this doesn't work is that the exception stack trace carries frame objects that reference local variables, so until that gets cleared the keys remain in scope. You can fix this by calling sys.exc_clear() before the gc.collect().\n",
"\n",
"Note that this problem is not present in Python 3: exc_info is automatically cleared when you exit the exception handler."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Partially ordered sets"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 52,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"{1} <= {2}"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 53,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"{1} >= {2}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The ordering relation on sets is defined to be subset inclusion, which is only a partial order rather than a total order. In particular this means that (x < y) is not the same as not (y <= x):"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 54,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"{1} < {2}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that this means that sorting lists of sets will not work correctly:"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[{-1, 1}, {0}, {1}]"
]
},
"execution_count": 55,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sorted([{-1, 1}, {0}, {1}])"
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 56,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"{1} < {-1, 1}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Because sets are not obeying the contract of ordering, they do not sort correctly and you get an allegedly sorted array with the final element strictly less than the first."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.10"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
@erthalion
Copy link

Thanks for sharing. There is one inconsistency, which is my favorite)

In [1]: a = 1

In [2]: b = 1

In [3]: a is b
Out[3]: True

In [4]: a = 1000

In [5]: b = 1000

In [6]: a is b
Out[6]: False

@DRMacIver
Copy link
Author

Oh yes, that one! It gets better though:

>>> def foo():
...     a = 1000
...     b = 1000
...     return a is b
... 
>>> foo()
True

The reason is that the constants used here are stored in the function object, where they're deduplicated:

>>> foo.__code__.co_consts
(None, 1000)

So inside the function both the literals are looked up as the same constant.

There is basically zero behaviour you can count on for when two numbers are going to be reference equal in Python.

@kxepal
Copy link

kxepal commented Dec 16, 2015

Similar things are not always similar.

Python 3.4.3 (default, Nov 12 2015, 20:43:56) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> a, b = b'foo', b'foo'
>>> a is b
True
>>> a, b = 'foo', 'foo'
>>> a is b
True
>>> a = 'foo'
>>> b = 'foo'
>>> a is b
True
>>> a = b'foo'
>>> b = b'foo'
>>> a is b
False

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment