Arsham Eslami asarama
asarama
/ request-body.json
Created
February 5, 2025 01:48
— forked from ianmcook/request-body.json
Use the Snowflake SQL REST API from a shell script with curl and jq to execute a query and download the result partitions in Arrow format
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "statement": "SELECT * FROM MYTABLE", | |
| "resultSetMetaData": { | |
| "format": "arrowv1" | |
| }, | |
| "timeout": 60, | |
| "database": "MYDATABASE", | |
| "schema": "MYSCHEMA", | |
| "warehouse": "MYWAREHOUSE", | |
| "role": "MYROLE" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from pyiceberg.catalog import load_catalog | |
| from pyiceberg.schema import Schema, NestedField | |
| from pyiceberg.types import ( | |
| StringType, | |
| LongType, | |
| TimestampType | |
| ) | |
| # Create the catalog connection |
asarama
/ convert_timestamps_to_iceberg_format.py
Created
December 1, 2024 20:33
Quick and dirty function to convert timestamps in a PyArrow data frame to Iceberg-compatible timestamps.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import pyarrow as pa | |
| from pyarrow import Table as ArrowTable | |
| def convert_dataframe_timestamps(data: ArrowTable) -> ArrowTable: | |
| """Convert timestamp columns to Iceberg-compatible timestamp format. | |
| This function processes a PyArrow table and converts timestamp columns to be compatible | |
| with Apache Iceberg specifications by: | |
| 1. Converting nanosecond precision timestamps to microsecond precision | |
| 2. Standardizing timezone-aware timestamps to UTC |