Skip to content

Instantly share code, notes, and snippets.

@jaraco
Last active August 29, 2015 14:16
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jaraco/d979a558bc0bf2194c23 to your computer and use it in GitHub Desktop.
Save jaraco/d979a558bc0bf2194c23 to your computer and use it in GitHub Desktop.
FixedOffset serialization fails
"""
Save this file as test_fo.py and run it
"""
import pprint
import json
import datetime
import pickle
jsonpickle = None
bson = None
def setup_module(module):
module.jsonpickle = __import__('jsonpickle')
module.bson = __import__('bson.tz_util')
def test_FixedOffsetSerializable():
fo = bson.tz_util.FixedOffset(-60*5, 'EST')
serialized = jsonpickle.dumps(fo)
pprint.pprint(json.loads(serialized))
restored = jsonpickle.loads(serialized)
print(restored._FixedOffset__offset)
assert vars(restored) == vars(fo)
def test_timedelta():
td = datetime.timedelta(-1, 68400)
serialized = jsonpickle.dumps(td)
pprint.pprint(json.loads(serialized))
restored = jsonpickle.loads(serialized)
assert restored == td
def test_stdlib_pickle():
fo = bson.tz_util.FixedOffset(-60*5, 'EST')
serialized = pickle.dumps(fo)
pprint.pprint(serialized)
restored = pickle.loads(serialized)
print(restored._FixedOffset__offset)
assert vars(restored) == vars(fo)
class FixedOffset(datetime.tzinfo):
def __init__(self, offset):
self.offset = datetime.timedelta(offset)
def __getinitargs__(self):
return self.offset,
def utcoffset(self, dt):
return self.offset
def tzname(self, dt):
return 'name'
def dst(self, dt):
return datetime.timedelta(0)
def test_nested_objects():
o = FixedOffset(99)
serialized = jsonpickle.dumps(o)
pprint.pprint(json.loads(serialized))
restored = jsonpickle.loads(serialized)
assert restored.offset == datetime.timedelta(99)
def test_datetime_with_fixed_offset():
fo = FixedOffset(-60)
dt = datetime.datetime.now().replace(tzinfo=fo)
serialized = jsonpickle.dumps(dt)
pprint.pprint(json.loads(serialized))
restored = jsonpickle.loads(serialized)
assert restored == dt
setup_params = dict(
install_requires=[
'jsonpickle',
'pymongo',
],
setup_requires=[
'pytest_runner',
],
tests_require=[
'pytest',
],
)
if __name__ == '__main__':
import sys
sys.argv[1:1] = ['pytest']
__import__('setuptools').setup(**setup_params)
@davvid
Copy link

davvid commented Mar 15, 2015

I've started taking a look. I also noticed that test_handles_cyclical_objects was broken by d7f84a27cac0dc94fdb705a0f8e13dbf240069b2 as well. It tests tuples that contain cyclical references.

One hole in the referencing scheme that has always been an especially tricky problem are tuples, because they are immutable. The hard part is that we need to make a reference to the object available so that restoring the contents of the tuple will allow those objects to create cycles back to the actual object. Thus, the tuple must exist before its items, which is a paradox since the empty tuple can never becomes a tuple-with-items and keep the same identity.

This is hole too strong for the proxy object scheme because if a tuple ends up containing a proxy then we can't swap it out later.

But, I think there is one gnarly and magical hack that could be the basis for how to deal with tuple's immutability. Instead of returning tuples, we could return a subclass of a tuple that is duck-type enough like a tuple that, as long as we document it, might just be good enough for the 90% use case.

The trick is as follows. The reason lists are not a problem is that we can create a list, create its contents, and then extend the list with the content because a list is mutable. If we could do the same with a tuple then there's no problem.

So how can we create an object that behaves like a list, and then later is swapped in-place to becomes a tuple that prevents item assignment? This is kinda crazy, and is definitely in the stupid python tricks category, but I guess it actually has a use case here...


class List(list):
    pass

class Tuple(tuple):
    pass

# Let's create an object that we can modify, and then coerce it into becoming tuple
xenomorph = List()
# It can even reference itself
xenomorph.extend([xenomorph, xenomorph])
# And then it becomes a tuple...
xenomorph.__class__ = Tuple

I'll need to rework that test case, but this just might work, so I'm going to try that. I have some preliminary cleanup stuff that I'll push out shortly, then I'll be going down this rabbit hole. Hopefully it works, we'll see...

@davvid
Copy link

davvid commented Mar 15, 2015

Ugh, nevermind, that totally doesn't work.. I'll keep digging. I was misled when I wrote some test code that made me believe that the above works, but it doesn't. doh.

TypeError: __class__ assignment: 'list' object layout differs from 'tuple'

@davvid
Copy link

davvid commented Mar 15, 2015

Going to try something like this... https://mail.python.org/pipermail/python-dev/2003-January/032043.html we'll see.

@davvid
Copy link

davvid commented Mar 15, 2015

# crash-python.py
class Tuple(tuple):
    __slots__ = ()

class List(list):
    __slots__ = ()

t = List()
t.__class__ = Tuple
# --- >8 --- >8 ---

$ python2.7 crash-python.py
Segmentation fault: 11

$ python3.4 crash-python.py
TypeError: __class__ assignment: 'List' object layout differs from 'Tuple'

Oh well, I'll probably try a different approach, like maybe returning an instance that behaves like a tuple, but without actually being an instance. That's unfortunate, though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment