Skip to content

Instantly share code, notes, and snippets.

doing open source

Valentin Haenel esc

doing open source
Block or report user

Report or block esc

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
View np vs. loop.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
esc / gulpio_results.txt
Created Sep 8, 2017
GulpIO Blogpost GulpIO Loader results
View gulpio_results.txt
(torch) root@7396071ace7e:/home/rgoyal/GulpIO-benchmarks# CUDA_VISIBLE_DEVICES=0,1 python --config configs/config_gulpio.json
=> active GPUs: 0,1
=> Output folder for this run -- jester_conv_example
> Using 10 processes for data loader.
> Training is getting started...
> Training takes 999999 epochs.
> Current LR : 0.001
Epoch: [0][0/1852] Time 12.410 (12.410) Data 9.079 (9.079) Loss 3.2856 (3.2856) Prec@1 9.375 (9.375) Prec@5 17.188 (17.188)
Epoch: [0][100/1852] Time 0.457 (0.854) Data 0.000 (0.429) Loss 2.5770 (3.0944) Prec@1 26.562 (12.655) Prec@5 68.750 (37.036)
Epoch: [0][200/1852] Time 0.456 (0.818) Data 0.000 (0.432) Loss 2.6250 (2.8789) Prec@1 25.000 (18.043) Prec@5 56.250 (47.528)
esc / jpeg_results.txt
Created Sep 8, 2017
GulpIO Blogpost JPEG Loader results
View jpeg_results.txt
(torch) root@7396071ace7e:/home/rgoyal/GulpIO-benchmarks# CUDA_VISIBLE_DEVICES=0,1 python --config configs/config_jpeg.json -g 0,1
=> active GPUs: 0,1
=> Output folder for this run -- jester_conv_example
> Using 10 processes for data loader.
> Training is getting started...
> Training takes 999999 epochs.
> Current LR : 0.001
Epoch: [0][0/1852] Time 77.101 (77.101) Data 73.654 (73.654) Loss 3.3249 (3.3249) Prec@1 0.000 (0.000) Prec@5 21.875 (21.875)
Epoch: [0][100/1852] Time 36.009 (7.519) Data 35.798 (7.183) Loss 2.6882 (3.0521) Prec@1 21.875 (13.784) Prec@5 54.688 (40.161)
Epoch: [0][200/1852] Time 13.762 (6.756) Data 13.562 (6.433) Loss 2.5396 (2.8169) Prec@1 20.312 (19.652) Prec@5 62.500 (50.451)
esc /
Last active Sep 8, 2017
GulpIO Blogpost Ingestor Example
adapter = Custom20BNCsvJpegAdapter(input_csv, jpeg_path, output_folder,
ingestor = GulpIngestor(adapter, output_folder, videos_per_chunk,
esc /
Last active Sep 8, 2017
GulpIO Blogpost Adapter
class AbstractDatasetAdapter(ABC):
""" Base class adapter for gulping (video) datasets.
Inherit from this class and implement the `iter_data` method. This method
should iterate over your entire dataset and for each element return a
dictionary with the following fields:
id : a unique(?) ID for the element.
frames : a list of frames (PIL images, numpy arrays..)
meta : a dictionary with arbitrary metadata (labels, start_time...)
For examples, see the custom adapters below.
esc /
Created Sep 8, 2017
GulpIO Blogpost Dataloader Example
from gulpio.dataset import GulpImageDataset
from gulpio.loader import DataLoader
from gulpio.transforms import Scale, CenterCrop, Compose, UnitNorm
# define data augmentations. Notice that there are different functions for videos and images
transforms = Compose([
Scale(28), # resize image by the shortest edge
UnitNorm(), # instance wise mean and std norm
esc /
Last active Sep 8, 2017
GulpIO Blogpost Reading Example
# import the main interface for reading
from gulpio import GulpDirectory
# instantiate the GulpDirectory
gulp_directory = GulpDirectory('/tmp/something_something_gulps')
# iterate over all chunks
for chunk in gulp_directory:
# for each 'video' get the metadata and all frames
for frames, meta in chunk:
# do something with the metadata
for i, f in enumerate(frames):
esc /
Last active Sep 29, 2016
Expose a bug in moto EC2 tag filtering
import boto3
import moto
def expose():
ec2 = boto3.resource('ec2')
blue, green = ec2.create_instances(
ImageId='ANY_ID', MinCount=2, MaxCount=2)
esc / gist:1f7097d6c33b037b6911
Created Apr 1, 2015
hashing with git and python
View gist:1f7097d6c33b037b6911
zsh» git init example
Initialized empty Git repository in /tmp/example/.git/
zsh» cd example
zsh» cp ../objektmodell-beispiel/ .
zsh» cat
#! /usr/bin/env python
""" Hello World! """
print 'Hello World!'
esc /
Created Oct 24, 2014
Testing LZ4 failure with blosc
import blosc
data = [
0x93, 0xb0, 0x49, 0xaf, 0x62, 0xad, 0xe3, 0xaa, 0xe4, 0xa5, 0x43, 0x20,
0x24, 0x29, 0xc9, 0xaf, 0xee, 0xad, 0x0b, 0xac, 0x3d, 0xa8, 0x1f, 0x99,
0x53, 0x27, 0xb6, 0x2b, 0x16, 0xb0, 0x5f, 0xae, 0x89, 0xac, 0x51, 0xa9,
0xfc, 0xa1, 0xc9, 0x24, 0x59, 0x2a, 0x2f, 0x2d, 0xb4, 0xae, 0xeb, 0xac,
0x2f, 0xaa, 0xec, 0xa4, 0x53, 0x21, 0x31, 0x29, 0x8f, 0x2c, 0x8e, 0x2e,
0x31, 0xad, 0xd6, 0xaa, 0x6d, 0xa6, 0xad, 0x1b, 0x3e, 0x28, 0x0a, 0x2c,
0xfd, 0x2d, 0xf8, 0x2f, 0x45, 0xab, 0x81, 0xa7, 0x1f, 0x95, 0x02, 0x27,
You can’t perform that action at this time.