Skip to content

Instantly share code, notes, and snippets.

Avatar
🌱
Eating plants

Alexander Galea agalea91

🌱
Eating plants
View GitHub Profile
@agalea91
agalea91 / ffmpeg_mp3_join.sh
Created Aug 13, 2021
ffmpeg audio file join (concatenate)
View ffmpeg_mp3_join.sh
ffmpeg -i "concat:file 1.mp3|file 2.mp3|file 3.mp3|..." -acodec copy ../Output-File.mp3
View redshift_database_table_transfer.md

Unload to s3

unload ('select * from schema.table') 
to 's3://path/to/s3/folder/'
-- iam_role 'arn:aws:iam::<aws-account-id>:role/<role-name>'
-- credentials 'aws_access_key_id=<key>;aws_secret_access_key=<secret>'
allowoverwrite
format as csv;
@agalea91
agalea91 / sqlalchemy_execute_raw_sql.py
Last active Feb 19, 2021
Run raw SQL and read/write from SQL database with Pandas using Sqlalchemy
View sqlalchemy_execute_raw_sql.py
def get_engine():
import sqlalchemy
import os
# os.environ["DB_URL"] = "postgresql://USER:PASS@HOST:PORT/DATABASE"
engine = sqlalchemy.create_engine(os.environ["DB_URL"])
return engine
def close_engine(engine):
engine.dispose()
@agalea91
agalea91 / ssh_pd_read_csv.py
Last active Jan 5, 2021
Read CSV file on remote server over SSH, using local system RSA auth
View ssh_pd_read_csv.py
SSH_HOST = ""
SSH_USER = ""
SSH_FILE_PATH = ""
COMPRESSION = "zip"
from paramiko import SSHClient
client = SSHClient()
client.load_system_host_keys()
client.connect(SSH_HOST, username=SSH_USER)
@agalea91
agalea91 / s3_gs_bigquery_upload.sh
Last active Oct 28, 2020
Upload from s3 to gs and then to bigquery
View s3_gs_bigquery_upload.sh
aws s3 cp s3://my_bucket/my_table /tmp/my_table --recursive \
&& gcloud config set account <name>@<project>.iam.gserviceaccount.com \
&& gcloud config set project <project> \
&& gcloud auth activate-service-account <name>@<project>.iam.gserviceaccount.com --key-file /my_path_to/google_credentials.json
&& gsutil cp -r /tmp/my_table gs://bucket_name/my_table \
&& bq mk --dataset my_dataset \
&& bq mk --table --schema my_table_schema.json my_dataset.my_table \
&& bq query --nouse_legacy_sql 'delete from my_dataset.my_table where <date_range_where_clause>;' \
&& bq load --source_format=CSV --quote "" -F='\t' my_dataset.my_table gs://bucket_name/my_table/0000_part_00 \
&& bq load --source_format=CSV --quote "" -F='\t' my_dataset.my_table gs://bucket_name/my_table/0001_part_00 \
@agalea91
agalea91 / redshift_unload_to_s3.sql
Last active Oct 24, 2020
Redshift unload to s3
View redshift_unload_to_s3.sql
unload ('
select * from my_table_name
where <date_range_where_clause>
')
to 's3://my-bucket/my_table_name/'
credentials 'aws_access_key_id=<key_id>;aws_secret_access_key=<secret_key>'
delimiter '\t'
allowoverwrite
;
@agalea91
agalea91 / fixed_dynamic_dag.py
Last active Sep 17, 2020
For medium blog post: Airflow Dynamic DAGs - Python Globals
View fixed_dynamic_dag.py
for symbol in ("BTC", "ETH", "LTC", "XLM"):
dag = create_dag(symbol=symbol)
globals()["{}_dag".format(symbol.lower())] = dag
@agalea91
agalea91 / broken_dynamic_dag.py
Last active Sep 17, 2020
For medium blog post: Airflow Dynamic DAGs - Python Globals
View broken_dynamic_dag.py
def create_dag(symbol):
with DAG(
"email_{}_price".format(symbol.lower()),
default_args={"start_date": "2020-01-01"},
schedule_interval="0 0 * * *",
) as dag:
get_price_task = PythonOperator(
task_id="get_price",
python_callable=get_price,
op_kwargs=dict(
@agalea91
agalea91 / static_dag.py
Last active Sep 17, 2020
For medium blog post: Airflow Dynamic DAGs - Python Globals
View static_dag.py
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from crypto_prices import get_price, send_email
with DAG(
"email_bitcoin_price",
default_args={"start_date": "2020-01-01"},
schedule_interval="0 0 * * *",
) as dag:
@agalea91
agalea91 / github_deploy_key_setup.md
Created Jun 24, 2020
How to setup a deploy key for repo on shared server environment
View github_deploy_key_setup.md

GitHub Deploy Key Setup

  1. Generate key
ssh-keygen
...
cp ./key-name* ~/.ssh
  1. Go to GitHub repo and add key-name.pub