Skip to content

Instantly share code, notes, and snippets.

View kawata-atsushi's full-sized avatar

KAWATA Atsushi kawata-atsushi

  • Classmethod, Inc.
  • Tokyo
  • 02:25 (UTC +09:00)
View GitHub Profile
@kawata-atsushi
kawata-atsushi / iceberg_migration_ctas.py
Created March 22, 2024 13:11
sample PySpark Script. Migration to Iceberg table using ctas.
from pyspark.sql import SparkSession
catalog_name = "glue_catalog"
bucket_name = "stage-ap-northeast-1-cm-zunda-demo"
warehouse_path = f"s3://{bucket_name}/iceberg"
spark = SparkSession.builder \
.config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
.config(f"spark.sql.catalog.{catalog_name}", "org.apache.iceberg.spark.SparkCatalog") \
.config(f"spark.sql.catalog.{catalog_name}.warehouse", warehouse_path) \
@kawata-atsushi
kawata-atsushi / iceberg_migration_add_files.py
Created March 22, 2024 09:36
sample PySpark Script. Migration to Iceberg table using add_files procedure.
from pyspark.sql import SparkSession
catalog_name = "glue_catalog"
bucket_name = "stage-ap-northeast-1-cm-zunda-demo"
warehouse_path = f"s3://{bucket_name}/iceberg"
spark = SparkSession.builder \
.config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
.config(f"spark.sql.catalog.{catalog_name}", "org.apache.iceberg.spark.SparkCatalog") \
.config(f"spark.sql.catalog.{catalog_name}.warehouse", warehouse_path) \
@kawata-atsushi
kawata-atsushi / sample-bash-script.sh
Created March 22, 2024 01:09
sample bash script. (Minimal safe Bash script template)
#!/usr/bin/env bash
set -Eeuo pipefail
function usage() {
cat <<-EOF
Usage: $(basename "${BASH_SOURCE[0]}") [-h] [-v] [-f] -p param_value arg1 [arg2...]
Script description here.
Available options: