This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
LOAD DATA LOCAL INFILE '<FILENAME>' | |
REPLACE | |
INTO TABLE <TABLENAME> | |
FIELDS TERMINATED BY ',' | |
ENCLOSED BY '"' | |
LINES TERMINATED BY '\n' | |
IGNORE 1 ROWS; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Write to staging file | |
with open(TEMP_FILE, 'w') as f: | |
writer = csv.DictWriter(f, fieldnames=HEADERS), | |
quoting=csv.QUOTE_ALL, | |
lineterminator='\n') | |
writer.writeheader() | |
for data in staging_rows: | |
writer.writerow(data) | |
print('Written to {}'.format(TEMP_FILE)) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import boto3 | |
import csv | |
import json | |
import os | |
import pymysql | |
import sys | |
from os.path import join, dirname | |
# Load environment settings if exists |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
@media (prefers-color-scheme: dark) { | |
body .gist .highlight { | |
background: #272822; | |
} | |
body .gist .blob-num, | |
body .gist .blob-code-inner, | |
body .gist .pl-s2, | |
body .gist .pl-stj { | |
color: #f8f8f2; | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[Unit] | |
Description=Airflow webserver daemon | |
After=network.target | |
Wants=postgresql.target | |
[Service] | |
Environment="PATH=/home/airflow/.local/bin" | |
User=airflow | |
Group=airflow | |
Type=simple |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[Unit] | |
Description=Airflow scheduler daemon | |
After=network.target | |
Wants=postgresql.target | |
[Service] | |
Environment="PATH=/home/airflow/.local/bin" | |
User=airflow | |
Group=airflow | |
Type=simple |