Skip to content

Instantly share code, notes, and snippets.

View esc's full-sized avatar
:octocat:
doing open source

Emergency Self-Construct esc

:octocat:
doing open source
385 followers · 91 following
View GitHub Profile
@esc
esc / np vs. loop.ipynb
Created May 17, 2019 08:43
np vs. loop
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@esc
esc / anaconda.sh
Last active October 8, 2018 21:06
Anconda activation and deactivation functions for ZSH
# Functions to activate/deactivate Continuum Analytics Anaconda Python distribution
# by manipulating the $PATH.
export ANACONDA_PATH="$HOME/anaconda/bin"
function have_anaconda(){
[[ -n $path[(r)$ANACONDA_PATH] ]]
}
function anaconda_on(){
if have_anaconda ; then
@esc
esc / gulpio_results.txt
Created September 8, 2017 12:13
GulpIO Blogpost GulpIO Loader results
(torch) root@7396071ace7e:/home/rgoyal/GulpIO-benchmarks# CUDA_VISIBLE_DEVICES=0,1 python train_gulp.py --config configs/config_gulpio.json
=> active GPUs: 0,1
=> Output folder for this run -- jester_conv_example
> Using 10 processes for data loader.
> Training is getting started...
> Training takes 999999 epochs.
> Current LR : 0.001
Epoch: [0][0/1852] Time 12.410 (12.410) Data 9.079 (9.079) Loss 3.2856 (3.2856) Prec@1 9.375 (9.375) Prec@5 17.188 (17.188)
Epoch: [0][100/1852] Time 0.457 (0.854) Data 0.000 (0.429) Loss 2.5770 (3.0944) Prec@1 26.562 (12.655) Prec@5 68.750 (37.036)
Epoch: [0][200/1852] Time 0.456 (0.818) Data 0.000 (0.432) Loss 2.6250 (2.8789) Prec@1 25.000 (18.043) Prec@5 56.250 (47.528)
@esc
esc / jpeg_results.txt
Created September 8, 2017 12:11
GulpIO Blogpost JPEG Loader results
(torch) root@7396071ace7e:/home/rgoyal/GulpIO-benchmarks# CUDA_VISIBLE_DEVICES=0,1 python train_jpeg.py --config configs/config_jpeg.json -g 0,1
=> active GPUs: 0,1
=> Output folder for this run -- jester_conv_example
> Using 10 processes for data loader.
> Training is getting started...
> Training takes 999999 epochs.
> Current LR : 0.001
Epoch: [0][0/1852] Time 77.101 (77.101) Data 73.654 (73.654) Loss 3.3249 (3.3249) Prec@1 0.000 (0.000) Prec@5 21.875 (21.875)
Epoch: [0][100/1852] Time 36.009 (7.519) Data 35.798 (7.183) Loss 2.6882 (3.0521) Prec@1 21.875 (13.784) Prec@5 54.688 (40.161)
Epoch: [0][200/1852] Time 13.762 (6.756) Data 13.562 (6.433) Loss 2.5396 (2.8169) Prec@1 20.312 (19.652) Prec@5 62.500 (50.451)
@esc
esc / gulpio_adapter_example.py
Last active September 8, 2017 12:00
GulpIO Blogpost Adapter
class AbstractDatasetAdapter(ABC):
""" Base class adapter for gulping (video) datasets.
Inherit from this class and implement the `iter_data` method. This method
should iterate over your entire dataset and for each element return a
dictionary with the following fields:
id : a unique(?) ID for the element.
frames : a list of frames (PIL images, numpy arrays..)
meta : a dictionary with arbitrary metadata (labels, start_time...)
For examples, see the custom adapters below.
"""
@esc
esc / gulpio_ingestor_example.py
Last active September 8, 2017 11:59
GulpIO Blogpost Ingestor Example
adapter = Custom20BNCsvJpegAdapter(input_csv, jpeg_path, output_folder,
shuffle=shuffle,
frame_size=img_size,
shm_dir_path=shm_dir
)
ingestor = GulpIngestor(adapter, output_folder, videos_per_chunk,
num_workers=num_workers)
ingestor()
@esc
esc / gulpio_dataloader_example.py
Created September 8, 2017 11:54
GulpIO Blogpost Dataloader Example
from gulpio.dataset import GulpImageDataset
from gulpio.loader import DataLoader
from gulpio.transforms import Scale, CenterCrop, Compose, UnitNorm
# define data augmentations. Notice that there are different functions for videos and images
transforms = Compose([
Scale(28), # resize image by the shortest edge
CenterCrop(28),
UnitNorm(), # instance wise mean and std norm
])
@esc
esc / gulpio_reading_example.py
Last active September 8, 2017 11:52
GulpIO Blogpost Reading Example
# import the main interface for reading
from gulpio import GulpDirectory
# instantiate the GulpDirectory
gulp_directory = GulpDirectory('/tmp/something_something_gulps')
# iterate over all chunks
for chunk in gulp_directory:
# for each 'video' get the metadata and all frames
for frames, meta in chunk:
# do something with the metadata
for i, f in enumerate(frames):
@esc
esc / expose_moto_bug.py
Last active September 29, 2016 08:39
Expose a bug in moto EC2 tag filtering
import boto3
import moto
@moto.mock_ec2
def expose():
ec2 = boto3.resource('ec2')
blue, green = ec2.create_instances(
ImageId='ANY_ID', MinCount=2, MaxCount=2)
ec2.create_tags(Resources=[blue.instance_id],
@esc
esc / notes.rst
Created September 25, 2013 18:04
Git server side software.