Skip to content

Instantly share code, notes, and snippets.

@Multihuntr
Multihuntr / accumulate_grads.py
Last active August 10, 2018 08:25
Accumulating gradients to reduce memory requirement per forward pass (using MNIST)
import numpy as np
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
def simple_model(input):
# This ensures that the model will always be instantiated the same, for comparison.
hidden_initializer = tf.constant_initializer(np.random.uniform(-0.025, 0.025, size=[784,100]))
hidden = tf.layers.dense(input, 100, kernel_initializer=hidden_initializer)
out_initializer = tf.constant_initializer(np.random.uniform(-0.025, 0.025, size=[100,10]))