Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
resource_gernerator
def resource_generator(row_processor, test_me=False):
"""A decorator to loop over all resources.
This convenience decorator is used inside processor modules to turn a
function that processes a single row into a function that returns a
generator of generators (all resources then all rows in each resource).
The decorator disables itself automatically in the context of pytests,
so that the decorated function can be tested more easily.
See `processors.parse_currency_fields module` for an example.
"""
def wrapper(resources, *args, **kwargs):
"""Return an resource iterator of row iterators"""
logging.info('args = %s', args)
logging.info('kwargs = %s', kwargs)
for resource in resources:
def process_rows(resource_):
for row in resource_:
yield row_processor(row, *args, **kwargs)
yield process_rows(resource)
if is_test() and not test_me:
return row_processor
else:
return wrapper
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment