Skip to content

Instantly share code, notes, and snippets.

View Multihuntr's full-sized avatar

Brandon Victor Multihuntr

  • La Trobe University
View GitHub Profile
@Multihuntr
Multihuntr / accumulate_grads_min.py
Created January 23, 2018 07:21
A minimal example of how you can accumulate gradients across batches, allowing you to train using much larger batch sizes than can fit in memory at the cost of speed.
import numpy as np
import tensorflow as tf
import sys
from tensorflow.examples.tutorials.mnist import input_data
n_pseudo_batches = int(sys.argv[1]) if len(sys.argv) > 1 else 128
actual_batch_size = int(sys.argv[2]) if len(sys.argv) > 2 else 32
iterations = int(sys.argv[3]) if len(sys.argv) > 3 else 10
@Multihuntr
Multihuntr / accumulate_grads.py
Last active August 10, 2018 08:25
Accumulating gradients to reduce memory requirement per forward pass (using MNIST)
import numpy as np
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
def simple_model(input):
# This ensures that the model will always be instantiated the same, for comparison.
hidden_initializer = tf.constant_initializer(np.random.uniform(-0.025, 0.025, size=[784,100]))
hidden = tf.layers.dense(input, 100, kernel_initializer=hidden_initializer)
out_initializer = tf.constant_initializer(np.random.uniform(-0.025, 0.025, size=[100,10]))
@Multihuntr
Multihuntr / random_queue.py
Created October 20, 2017 07:55
Blocking Thread-safe Random Queue
import queue
import random
class RandomQueue(queue.Queue):
def _put(self, item):
n = len(self.queue)
i = random.randint(0, n)
self.queue.insert(i, item)