vlandham

Here's a little walkthrough of how Yannick and I are using feature branches and pull requests to develop new features and adding them to the project. Below are the steps I take when working on a new feature. Hopefully this, along with watching the process on Github, will serve as a starting point to having everyone use a similar workflow.

Questions, comments, and suggestions for improvements welcome!

Start with the latest on master

When starting a new feature, I make sure to start with the latest and greatest codebase:

git checkout master

bartolsthoorn

import torch
import torch.nn as nn
import numpy as np
import torch.optim as optim
from torch.autograd import Variable

# (1, 0) => target labels 0+2
# (0, 1) => target labels 1
# (1, 1) => target labels 3
train = []

naotokui

# coding: utf8

## based on this article: http://qiita.com/mokemokechicken/items/937a82cfdc31e9a6ca12

import numpy as np
from keras.models import Sequential
from keras.engine.topology import Input, Container
from keras.engine.training import Model
from keras.layers.core import Dense

markito

How much memory is permanently in memory vs how much is used for transformations
(ratio)
spark.storage.memoryFraction

Suggested settings... (need to debug the logs after these settings)
-XX:+UseG1GC -XX:+PrintFlagsFinal -XX:+PrintReferenceGC -verbose:gc 
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintAdaptiveSizePolicy
-XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark 
-Xms88g -Xmx88g -XX:InitiatingHeapOccupancyPercent=35 
-XX:ConcGCThread=15 -XX:+AlwaysPreTouch

thomwolf

# input_seqs is a batch of input sequences as a numpy array of integers (word indices in vocabulary) padded with zeroas
input_seqs = Variable(torch.from_numpy(input_seqs.astype('int64')).long())

# First: order the batch by decreasing sequence length
input_lengths = torch.LongTensor([torch.max(input_seqs[i, :].data.nonzero()) + 1 for i in range(input_seqs.size()[0])])
input_lengths, perm_idx = input_lengths.sort(0, descending=True)
input_seqs = input_seqs[perm_idx][:, :input_lengths.max()]

# Then pack the sequences
packed_input = pack_padded_sequence(input_seqs, input_lengths.cpu().numpy(), batch_first=True)

MattKleinsmith

# https://gist.github.com/kevinzakka/d33bf8d6c7f06a9d8c76d97a7879f5cb#file-data_loader-py
# This is an example for the MNIST dataset (formerly CIFAR-10).
# There's a function for creating a train and validation iterator.
# There's also a function for creating a test iterator.
# Inspired by https://discuss.pytorch.org/t/feedback-on-pytorch-for-kaggle-competitions/2252/4

# Adapted for MNIST by github.com/MatthewKleinsmith

import numpy as np
import torch

ruslangrimov

import tensorflow as tf
from tensorflow.python.client import timeline
from keras import backend as K

run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
run_metadata = tf.RunMetadata()

model = ...  # A Keras model

fn = K.function(model.inputs, model.outputs, options=run_options, run_metadata=run_metadata)

mattiasarro

Tensorflow 1.6 on macOS High Sierra 10.13.3 with GPU Acceleration (without disabling SIP)

This gist (based on a blog post at byai.io) documents how to set up TensorFlow 1.6 with (e)GPU support without the need to disable SIP. Following the original gist got me a saystem in which training TF on eGPU was successful, but there were various visual glitches due to the newer / less stable version of the driver.

As pointed out by ronchigram, many people are having issues with newer NVIDIA drivers, so it's worth using the nvidia-update script by Benjamin Dobell that installs the latest stable NVIDIA web driver, and if necessary patches it to run on your system. We also don't need to disable SIP when using nvidia-update.

I have als

jeremyjordan

from keras.callbacks import Callback
import keras.backend as K
import numpy as np

class SGDRScheduler(Callback):
    '''Cosine annealing learning rate scheduler with periodic restarts.

    # Usage
        ```python
            schedule = SGDRScheduler(min_lr=1e-5,

thomwolf

model.zero_grad()                                   # Reset gradients tensors
for i, (inputs, labels) in enumerate(training_set):
    predictions = model(inputs)                     # Forward pass
    loss = loss_function(predictions, labels)       # Compute loss function
    loss = loss / accumulation_steps                # Normalize our loss (if averaged)
    loss.backward()                                 # Backward pass
    if (i+1) % accumulation_steps == 0:             # Wait for several backward steps
        optimizer.step()                            # Now we can do an optimizer step
        model.zero_grad()                           # Reset gradients tensors
        if (i+1) % evaluation_steps == 0:           # Evaluate the model when we...