Skip to content

Instantly share code, notes, and snippets.

@phraniiac
Last active July 22, 2017 11:44
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save phraniiac/3743d615174fe301c6df2d4f56884e32 to your computer and use it in GitHub Desktop.
Save phraniiac/3743d615174fe301c6df2d4f56884e32 to your computer and use it in GitHub Desktop.
def process_and_insert(batch):
for tablename in tablenams:
# some processing for each typeof table.
db.insert(batch, tablename)
def insert_function(resultiterator, tablenames):
for batch in iterator_function(resultiterator):
process_and_insert(batch, tablenames)
def iterator_function(resultiterator):
list_for_batch_size = []
for obj in resultiterator:
list_for_batch_size.append(obj)
if len(list_for_batch_size) == BATCH_SIZE:
yield(list_for_batch_size)
# for left records after last iteration.
yield(list_of_batch_size)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment