Skip to content

Instantly share code, notes, and snippets.

@nvie

nvie/generic.py Secret

Created October 3, 2014 08:14
Show Gist options
  • Select an option

  • Save nvie/f304caf3b4f1ca4c3884 to your computer and use it in GitHub Desktop.

Select an option

Save nvie/f304caf3b4f1ca4c3884 to your computer and use it in GitHub Desktop.
Complete solution to the problem posed in this blog post: http://nvie.com/posts/modifying-deeply-nested-structures/
def traverse(obj, path=None, callback=None):
"""
Traverse an arbitrary Python object structure (limited to JSON data
types), calling a callback function for every element in the structure,
and inserting the return value of the callback as the new value.
"""
if path is None:
path = []
if isinstance(obj, dict):
value = {k: traverse(v, path + [k], callback)
for k, v in obj.items()}
elif isinstance(obj, list):
value = [traverse(elem, path + [[]], callback)
for elem in obj]
else:
value = obj
if callback is None:
return value
else:
return callback(path, value)
def traverse_modify(obj, target_path, action):
"""
Traverses an arbitrary object structure and where the path matches,
performs the given action on the value, replacing the node with the
action's return value.
"""
target_path = to_path(target_path)
def transformer(path, value):
if path == target_path:
return action(value)
else:
return value
return traverse(obj, callback=transformer)
def to_path(path):
"""
Helper function, converting path strings into path lists.
>>> to_path('foo')
['foo']
>>> to_path('foo.bar')
['foo', 'bar']
>>> to_path('foo.bar[]')
['foo', 'bar', []]
"""
if isinstance(path, list):
return path # already in list format
def _iter_path(path):
for parts in path.split('[]'):
for part in parts.strip('.').split('.'):
yield part
yield []
return list(_iter_path(path))[:-1]
from operator import itemgetter
from generic import traverse_modify
d = {
"timestamp": 1412282459,
"res": [
{
"group": "1",
"catlist": [
{
"cat": "1",
"start": "none",
"stop": "none",
"points": [
{"point": "1", "start": "13.00", "stop": "13.35"},
{"point": "2", "start": "11.00", "stop": "14.35"}
]
}
]
}
]
}
def sort_points(points):
"""Will sort a list of points."""
return sorted(points, reverse=True, key=itemgetter('stop'))
print(traverse_modify(d, 'res[].catlist[].points', sort_points))
@rdavey

rdavey commented Jun 20, 2018

Copy link
Copy Markdown

Nice code!

How would one go about using this code to modify each value in the dict based on a lookup of the key and return the new dict?

@peterwwillis

Copy link
Copy Markdown

@nvie I have a small bug fix here: https://gist.github.com/peterwwillis/3b14de8d2c7e9f6ce8899266d8aeea6d when the data structure is a list and its first element is a dict, it can insert an empty string at the beginning of the target_path which I couldn't match against, so I added a continue on empty parts of the path

@metinsenturk

Copy link
Copy Markdown

This gist is super awesome !

I was looking into ways of nested search and was even started a few tests on how I could make it into a package! I think, getting a value from a dict with dot syntax, like {}.get('animals.cats') can make our lives super easier.

I will work on my version of this idea and practice this gist a lot. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment