Skip to content

Instantly share code, notes, and snippets.

@jaklinger
Created May 11, 2018 13:46
Show Gist options
  • Save jaklinger/2b5b9c91c16dd9697848f94405796570 to your computer and use it in GitHub Desktop.
Save jaklinger/2b5b9c91c16dd9697848f94405796570 to your computer and use it in GitHub Desktop.
Execute SELECT statements in chunks with SqlAlchemy
from sqlalchemy import create_engine
from sqlalchemy import text as sql_text
# Stolen from https://stackoverflow.com/a/434328/1571593
def chunker(seq, size):
return (seq[pos:pos + size] for pos in range(0, len(seq), size))
# Execute {SELECT ... IN ...} in chunks
'''Notes:
1) query has to be of the form:
SELECT something FROM a_table WHERE something_else IN :values;
2) In the SELECT statement Don't change the name of ':values'
'''
def execute_IN_in_chunks(con, query, chunkable, chunk_size):
output = []
for chunk in chunker(chunkable, chunk_size):
result = con.execute(sql_text(query), values=tuple(chunk))
output += result.fetchall()
return output
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment