-
Go to https://developer.apple.com/downloads/index.action and search for "Command line tools" and choose the one for your Mac OSX
-
Go to http://brew.sh/ and enter the one-liner into the Terminal, you now have
brew
installed (a better Mac ports) -
Install transmission-daemon with
brew install transmission
-
Copy the startup config for launchctl with
ln -sfv /usr/local/opt/transmission/*.plist ~/Library/LaunchAgents
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#! /bin/bash | |
wget https://github.com/google/protobuf/releases/download/v2.6.1/protobuf-2.6.1.tar.gz | |
tar xzf protobuf-2.6.1.tar.gz | |
cd protobuf-2.6.1 | |
sudo apt-get update | |
sudo apt-get install build-essential | |
sudo ./configure | |
sudo make | |
sudo make check | |
sudo make install |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#! /bin/bash | |
sudo apt-get update -y | |
sudo apt-get upgrade -y | |
sudo apt-get install -y git htop tmux transmission-cli transmission-daemon | |
sudo apt-get -y update | |
sudo /etc/init.d/transmission-daemon stop |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package nodescala | |
import com.sun.net.httpserver._ | |
import scala.concurrent._ | |
import scala.concurrent.duration._ | |
import ExecutionContext.Implicits.global | |
import scala.async.Async.{async, await} | |
import scala.collection._ | |
import scala.collection.JavaConversions._ | |
import java.util.concurrent.{Executor, ThreadPoolExecutor, TimeUnit, LinkedBlockingQueue} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
version: '2' | |
services: | |
minio: | |
restart: always | |
image: docker.io/bitnami/minio:2021 | |
ports: | |
- '9000:9000' | |
environment: | |
- MINIO_ROOT_USER=miniokey |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# suppose my data file name has the following format "datatfile_YYYY_MM_DD.csv"; this file arrives in S3 every day. | |
file_suffix = "{{ execution_date.strftime('%Y-%m-%d') }}" | |
bucket_key_template = 's3://[bucket_name]/datatfile_{}.csv'.format(file_suffix) | |
file_sensor = S3KeySensor( | |
task_id='s3_key_sensor_task', | |
poke_interval=60 * 30, # (seconds); checking file every half an hour | |
timeout=60 * 60 * 12, # timeout in 12 hours | |
bucket_key=bucket_key_template, | |
bucket_name=None, | |
wildcard_match=False, |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from airflow import DAG | |
from airflow.operators.sensors import S3KeySensor | |
from airflow.operators import BashOperator | |
from datetime import datetime, timedelta | |
yday = datetime.combine(datetime.today() - timedelta(1), | |
datetime.min.time()) | |
default_args = { | |
'owner': 'msumit', |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
with DAG(**dag_config) as dag: | |
# Declare pipeline start and end task | |
start_task = DummyOperator(task_id='pipeline_start') | |
end_task = DummyOperator(task_id='pipeline_end') | |
for account_details in pipeline_config['task_details']['accounts']: | |
#Declare Account Start and End Task | |
if account_details['runable']: | |
acct_start_task = DummyOperator(task_id=account_details['account_id'] + '_start') | |
acct_start_task.set_upstream(start_task) |
Setup Parquet-tools
brew install parquet-tools
Help parquet-tools -h
parquet-tools rowcount part-00000-fc34f237-c985-4ebc-822b-87fa446f6f70.c000.snappy.parquet
parquet-tools head -n 1 part-00000-fc34f237-c985-4ebc-822b-87fa446f6f70.c000.snappy.parquet
parquet-tools meta part-00000-fc34f237-c985-4ebc-822b-87fa446f6f70.c000.snappy.parquet
NewerOlder