Skip to content

Instantly share code, notes, and snippets.

@bobbruno
Created November 10, 2015 00:16
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save bobbruno/4960c005248258a13169 to your computer and use it in GitHub Desktop.
Save bobbruno/4960c005248258a13169 to your computer and use it in GitHub Desktop.
python decorator for bufferising generator output
def bufferise(defbuf=20, defskip=0):
def decorate(function):
def wrapper(*args, **kwargs):
bufsize = kwargs['bufsize'] if 'bufsize' in kwargs else defbuf
skiplines = kwargs['skiplines'] if 'skiplines' in kwargs else defskip
print 'Bufsize = {}'.format(bufsize)
print 'Skip {} lines'.format(skiplines)
if skiplines:
for i, record in enumerate(function(*args, **kwargs), start=1):
if i > skiplines:
break
while True:
buffer = []
i = 0
for i, record in enumerate(function(*args, **kwargs), start = 1):
buffer.append(record)
if i > bufsize:
break
if i <= bufsize:
break
yield buffer
if len(buffer):
yield buffer
raise StopIteration
return wrapper
return decorate
@bobbruno
Copy link
Author

Bufferise

This code allows to easily bufferise the output of any generator. Just decorate the generator function with @bufferize, and you automatically get a buffer or 20 records. You can alternatively pass a defbuf parameter to the decorator, or even add an additional bufsize parameter at the call to the decorated function, to override the default.

The same decorator allows you to skip leading lines (return values) as well. The defskip defaults to 0, but can be set to any value at the decoration point. Alternatively, you can also pass an additional skiplines parameter to the decorated generator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment