Skip to content

Instantly share code, notes, and snippets.

@ayemos
Last active October 4, 2019 02:17
Show Gist options
  • Save ayemos/c70b3b129e27b00f93231e4ace85b2fb to your computer and use it in GitHub Desktop.
Save ayemos/c70b3b129e27b00f93231e4ace85b2fb to your computer and use it in GitHub Desktop.
from datetime import datetime, timedelta
import random
import awswrangler
import pandas as pd
df = pd.DataFrame(
{
'x': (random.random() for _ in range(1000)),
'date': (
(datetime.today() + timedelta(days=-5 * 365 + i)).date()
for i in range(1000))})
df['year'] = df['date'].apply(lambda d: d.year)
df['month'] = df['date'].apply(lambda d: d.month)
session = awswrangler.Session()
session.pandas.to_parquet(
df,
mode='overwrite',
path='s3://BUCKET_NAME/PREFIX/',
database='wrangler_db',
table='dummy_table_002',
partition_cols=['year', 'month'])
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment