Skip to content

Instantly share code, notes, and snippets.

@arihantsurana

arihantsurana/AthenaToDataFrameExample.py

Last active July 23, 2020 09:57
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save arihantsurana/7172e772e0622a25560aa0479b1cd123 to your computer and use it in GitHub Desktop.
Save arihantsurana/7172e772e0622a25560aa0479b1cd123 to your computer and use it in GitHub Desktop.
Download ZIP
An example for using hip_data_tools for Athena to Dataframe utility
Raw
AthenaToDataFrameExample.py
from hip_data_tools.aws.common import AwsConnectionSettings
from hip_data_tools.etl.athena_to_dataframe import AthenaToDataFrameSettings, AthenaToDataFrame
aws_conn = AwsConnectionSettings(
region="ap-southeast-2",
secrets_manager=None,
profile="default") # to access the default aws profile on your machine
etl = AthenaToDataFrame(
settings=AthenaToDataFrameSettings(
source_database="my_athena_database",
source_table="my_table",
source_connection_settings=aws_conn
)
)
df = etl.get_all_files_as_dataframe()
df.info()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment