Skip to content

Instantly share code, notes, and snippets.

View karenyyng's full-sized avatar

Karen Ng karenyyng

79 followers · 38 following
View GitHub Profile
@karenyyng
karenyyng / Galsim_pip_requirement.txt
Last active March 21, 2016 04:28
python PIP requirement list
numpy==1.10.4
astropy==0.4.2
scipy==0.14.0
pyyaml
starlink-pyast
@karenyyng
karenyyng / Failed_Galsim_Dockerfile
Last active March 21, 2016 22:58
Failed_Galsim_Dockerfile
# Should update the base image ubuntu version to 16.04 LTS when it comes out
FROM ubuntu:14.04
RUN echo "\n\n---------- Downloading and building all OS level tools-----\n\n"
RUN apt update
# -y flag bypasses all questions
RUN apt install -y \
build-essential \
software-properties-common \
python-dev \
@karenyyng
karenyyng / gs.error
Last active March 21, 2016 23:00
gs.error fail to build Galsim with Dockerfile
Unable to get python include path python executable:
/usr/bin/python
Using the following options:
CXX = g++
FLAGS =
EXTRA_FLAGS =
LINKFLAGS = ['-fopenmp']
DEBUG = True
EXTRA_DEBUG = False
@karenyyng
karenyyng / pydata_list.md
Last active April 7, 2016 03:28
Incomplete list of cool Python data-related projects

Personal list of cool Python related projects to play with. Projects are counted as long as they have a Python API regardless if the underlying code is written in another language.

Data analytics (pipelined) frameworks

  • PySpark
  • GraphLab Create
  • Scikit Learn
  • Tensorflow
  • Cloud Dataflow
@karenyyng
karenyyng / parquet-benchmark-20170210.py
Created March 20, 2017 21:40 — forked from wesm/parquet-benchmark-20170210.py
Parquet multithreaded benchmarks
import gc
import os
import time
import numpy as np
import pandas as pd
from pyarrow.compat import guid
import pyarrow as pa
import pyarrow.parquet as pq
import snappy
@karenyyng
karenyyng / using_remote_IPython_notebook_session.md
Last active May 5, 2017 19:47
using_remote_IPython_notebook_session.md

This post guides you through using an IPython notebook with a kernel running on a remote server / machine in the most painless way possible, even if your remote machine is behind a firewall.

This involves using your remote server / machine as a proxy. (Many thanks to Todd Gamblin for showing me how this life-changing trick works.) This trick is awesome in a sense that it reduces one layer of ssh port that you have to specify. With ssh remote forwarding method such as:

$ ssh -N -f -L $<LOCAL_PORT>:127.0.0.1:$<REMOTE_PORT> $REMOTE_HOST -vvv

you have to worry both about remote host actively listening to remote port and that the local port is available. With this proxy method, you can just treat yourself as sitting at the remote machine.

@karenyyng
karenyyng / ADS_Language_Query.ipynb
Created May 20, 2017 23:31 — forked from jakevdp/ADS_Language_Query.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@karenyyng
karenyyng / Sports1M C3D Network to Keras.ipynb
Created May 26, 2017 01:47 — forked from albertomontesg/Sports1M C3D Network to Keras.ipynb
C3D Model for Keras
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@karenyyng
karenyyng / load_conda.sh
Last active October 30, 2017 20:45
load_conda.sh
if [[ -z $CONDA_ENV ]]; then
echo "CONDA_ENV is not defined. Quitting installation script."
exit 1
fi
if [[ $NERSC_HOST == "cori"* ]]; then
DIR=/global/common/cori/software/python/3.5-anaconda
module load python/3.5-anaconda
echo 'On Cori: echo finish loading python/3.5-anaconda'
@karenyyng
karenyyng / example_conda_setup_script.sh
Last active December 26, 2017 21:58
Example project conda setup script
# -------------
# Author: Karen Ng <mailto:karen.yyng@gmail.com>
# dependency: wget
# usage:
# source THIS_SCRIPT.sh
# -------------
# set per project settings
export CONDA_ENV=general
export INTEL_PYTHON_VERSION=2018.0.0
export PY_VERSION=35