Skip to content

Instantly share code, notes, and snippets.

@samklr
samklr / install-proto.sh
Created April 20, 2015 08:19
Install Protobuf debian ...
#! /bin/bash
wget https://github.com/google/protobuf/releases/download/v2.6.1/protobuf-2.6.1.tar.gz
tar xzf protobuf-2.6.1.tar.gz
cd protobuf-2.6.1
sudo apt-get update
sudo apt-get install build-essential
sudo ./configure
sudo make
sudo make check
sudo make install
Optimizing Google’s Warehouse Scale Computers: The NUMA Experience - authors from Univ. of Cal (SD) & Google!
http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/40696.pdf
MegaPipe: A New Programming Interface for Scalable Network I/O by several authors!
https://docs.google.com/viewer?a=v&pid=forums&srcid=MDUxODU4OTA1NTU1MzUxODE5MDQBMTA5MDUzNTI5Mzg2ODk0MjY5NjUBYWM5cnR1MEY4Z1FKATQBAXYy
What Every Programmer Should Know About Memory by Ulrich Drepper
http://www.akkadia.org/drepper/cpumemory.pdf
Memory Barriers: a Hardware View for Software Hackers - Paul E. McKenney (Linux Technology Center - IBM Beaverton)
@samklr
samklr / seedex.sh
Last active December 3, 2021 16:50
Scripts to build a torrent seed box with transmission on ubuntu
#! /bin/bash
sudo apt-get update -y
sudo apt-get upgrade -y
sudo apt-get install -y git htop tmux transmission-cli transmission-daemon
sudo apt-get -y update
sudo /etc/init.d/transmission-daemon stop
@samklr
samklr / INSTALL.md
Created October 10, 2021 12:15 — forked from jpillora/INSTALL.md
Headless Transmission on Mac OS X
  1. Go to https://developer.apple.com/downloads/index.action and search for "Command line tools" and choose the one for your Mac OSX

  2. Go to http://brew.sh/ and enter the one-liner into the Terminal, you now have brew installed (a better Mac ports)

  3. Install transmission-daemon with

    brew install transmission
    
  4. Copy the startup config for launchctl with

    ln -sfv /usr/local/opt/transmission/*.plist ~/Library/LaunchAgents
    
package nodescala
import com.sun.net.httpserver._
import scala.concurrent._
import scala.concurrent.duration._
import ExecutionContext.Implicits.global
import scala.async.Async.{async, await}
import scala.collection._
import scala.collection.JavaConversions._
import java.util.concurrent.{Executor, ThreadPoolExecutor, TimeUnit, LinkedBlockingQueue}
version: '2'
services:
minio:
restart: always
image: docker.io/bitnami/minio:2021
ports:
- '9000:9000'
environment:
- MINIO_ROOT_USER=miniokey
# suppose my data file name has the following format "datatfile_YYYY_MM_DD.csv"; this file arrives in S3 every day.
file_suffix = "{{ execution_date.strftime('%Y-%m-%d') }}"
bucket_key_template = 's3://[bucket_name]/datatfile_{}.csv'.format(file_suffix)
file_sensor = S3KeySensor(
 task_id='s3_key_sensor_task',
 poke_interval=60 * 30, # (seconds); checking file every half an hour
 timeout=60 * 60 * 12, # timeout in 12 hours
 bucket_key=bucket_key_template,
 bucket_name=None,
 wildcard_match=False,
@samklr
samklr / s3_sensor.py
Created February 12, 2021 20:03 — forked from msumit/s3_sensor.py
Airflow file sensor example
from airflow import DAG
from airflow.operators.sensors import S3KeySensor
from airflow.operators import BashOperator
from datetime import datetime, timedelta
yday = datetime.combine(datetime.today() - timedelta(1),
datetime.min.time())
default_args = {
'owner': 'msumit',
with DAG(**dag_config) as dag:
# Declare pipeline start and end task
start_task = DummyOperator(task_id='pipeline_start')
end_task = DummyOperator(task_id='pipeline_end')
for account_details in pipeline_config['task_details']['accounts']:
#Declare Account Start and End Task
if account_details['runable']:
acct_start_task = DummyOperator(task_id=account_details['account_id'] + '_start')
acct_start_task.set_upstream(start_task)
@samklr
samklr / parquet_tools.md
Last active January 22, 2021 19:50
Parquet Tools

Setup Parquet-tools brew install parquet-tools

Help parquet-tools -h

parquet-tools rowcount part-00000-fc34f237-c985-4ebc-822b-87fa446f6f70.c000.snappy.parquet

parquet-tools head -n 1 part-00000-fc34f237-c985-4ebc-822b-87fa446f6f70.c000.snappy.parquet

parquet-tools meta part-00000-fc34f237-c985-4ebc-822b-87fa446f6f70.c000.snappy.parquet