Skip to content

Instantly share code, notes, and snippets.

View ethan-funny's full-sized avatar
:octocat:
Focusing

Ethan ethan-funny

:octocat:
Focusing
View GitHub Profile
@ethan-funny
ethan-funny / folder_splitter.py
Created April 18, 2018 14:43 — forked from zupo/folder_splitter.py
Split a folder with many files into subfolders with N files. Usage: python folder_splitter.py path/to/target/folder
# -*- coding: utf-8 -*-
# @author: Peter Lamut
import argparse
import os
import shutil
N = 10 # the number of files in seach subfolder folder
@ethan-funny
ethan-funny / bnlstm.py
Created December 18, 2017 07:23 — forked from spitis/bnlstm.py
Batch normalized LSTM Cell for Tensorflow
"""adapted from https://github.com/OlavHN/bnlstm to store separate population statistics per state"""
import tensorflow as tf, numpy as np
RNNCell = tf.nn.rnn_cell.RNNCell
class BNLSTMCell(RNNCell):
'''Batch normalized LSTM as described in arxiv.org/abs/1603.09025'''
def __init__(self, num_units, is_training_tensor, max_bn_steps, initial_scale=0.1, activation=tf.tanh, decay=0.95):
"""
* max bn steps is the maximum number of steps for which to store separate population stats
"""
@ethan-funny
ethan-funny / local_distributed_benchmark.py
Created September 13, 2017 03:29 — forked from yaroslavvb/local_distributed_benchmark.py
Benchmark distributed tensorflow locally by adding vector of ones on worker2 to variable on worker1 as fast as possible
"""Benchmark tensorflow distributed by adding vector of ones on worker2
to variable on worker1 as fast as possible.
On 2014 macbook, TensorFlow 0.10 this shows
Local rate: 2175.28 MB per second
Distributed rate: 107.13 MB per second
"""
import tensorflow as tf
import time
from datetime import datetime
import numpy as np
FLAGS = tf.app.flags.FLAGS
tf.app.flags.DEFINE_bool('cluster', False, '')
def log(obj):
print datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S.%f')[:-3], obj
@ethan-funny
ethan-funny / matmul_benchmark.py
Created September 13, 2017 03:09 — forked from ahaider3/matmul_benchmark.py
Distributed Tensorflow Overhead Measurement Benchmark
from __future__ import print_function
import sys
import os
import argparse
import numpy as np
import time
import tensorflow as tf
# single node cluster spec params
@ethan-funny
ethan-funny / sharded_ps_benchmark.py
Created September 11, 2017 07:32 — forked from yaroslavvb/sharded_ps_benchmark.py
Example of local cluster with multiple workers/training loops sharded parameter server
#!/usr/bin/env python
# Benchmark transferring data, part of troubleshooting https://github.com/tensorflow/tensorflow/issues/6116
#
# Take a independent workers communicating with b parameter shards
# Each worker tries to add to variables stored on parameter server as fast as
# possible.
#
# macbook
# ps=1: 1.6 GB/s
# ps=2: 2.6 GB/s
@ethan-funny
ethan-funny / tf_serving.sh
Created September 9, 2017 05:32 — forked from ericyue/tf_serving.sh
Install Tensorflow Serving on Centos 7 (CPU)
sudo su
# Java
yum -y install java-1.8.0-openjdk-devel
# Build Esentials (minimal)
yum -y install gcc gcc-c++ kernel-devel make automake autoconf swig git unzip libtool binutils
# Extra Packages for Enterprise Linux (EPEL) (for pip, zeromq3)
yum -y install epel-release
import datetime
import pytz
from tensorflow.contrib.session_bundle import exporter
from tensorflow.python.client import timeline
import glob
import json
import time
import math
import numpy as np
import os
wget http://mirror.sdunix.com/gnu/emacs/emacs-24.5.tar.gz
tar zxvfp emacs-24.5.tar.gz
cd emacs-24.5/
./configure
sudo yum -y install libXpm-devel libjpeg-turbo-devel openjpeg-devel openjpeg2-devel turbojpeg-devel giflib-devel libtiff-devel gnutls-devel libxml2-devel GConf2-devel dbus-devel wxGTK-devel gtk3-devel
./configure
make
sudo make install
@ethan-funny
ethan-funny / tmux_local_install.sh
Created June 23, 2017 01:57 — forked from ryin/tmux_local_install.sh
bash script for installing tmux without root access
#!/bin/bash
# Script for installing tmux on systems where you don't have root access.
# tmux will be installed in $HOME/local/bin.
# It's assumed that wget and a C/C++ compiler are installed.
# exit on error
set -e
TMUX_VERSION=1.8