Skip to content

Instantly share code, notes, and snippets.

@canimus
Created December 4, 2019 02:38
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save canimus/fc1fa208e2a1a6d355c27baa7042bf71 to your computer and use it in GitHub Desktop.
Save canimus/fc1fa208e2a1a6d355c27baa7042bf71 to your computer and use it in GitHub Desktop.
Parquet writer from dataframe
import pandas as pd
import numpy as np
import pymssql
conn = pymssql.connect("localhost", "SA", "yourStrong(!)Password", "FORCE_PAPPA")
with open("task.sql") as infile:
doc_query = infile.read().strip()
time_range = list(pd.date_range(start='2019-01-01', periods=12, freq="MS").strftime("%Y-%m-%d").values)
for k,v in enumerate(time_range):
if k == len(time_range) - 1: break
query = doc_query.format(time_range[k], time_range[k+1])
df = pd.read_sql(query, conn)
if df.shape[0] > 0: df.to_parquet("task_{}.parquet".format(k))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment