By Adam Anderson
Aggregated links to tutorials with some summaries. Like all of my Gist notes, this document is mostly so I can keep track of useful resources.
Table of Contents generated with DocToc
Tenorflow is a programming system in which you represent computations as graphs. This lets us represent computation without performing it until we decide to. Nodes in the graphs are called ops (operations). An op operates on tensors.
TensorFlow programs use a tensor data structure to represent all data. A tensor is just an n-dimensional array with static type and dynamic dimensions.
A tensor is described by three properties: Rank, Shape, and Type.
Rank - The number of dimensions in the tensor (how many subscript indices we need to access an element). A standard array is a rank 2 tensor, whose elements are accessed t[i, j]. A list is a rank 1 tensor. For a rank 3 tensor, elements are accessed t[i, j, k].
Shape - A list of numbers describing the shape of the tensor along each dimension. For example, the rank 2 tensor [[1, 2, 3], [4, 5, 6]] has shape [2, 3]. In general, the shape of a tensor is [D0, D1, ... Dn-1].
Type - Data type contained by the tensor. For example, tf.float32, tf.int8, andtf.bool.
In a TensorFlow program, we create a graph that does operations on tensors and then launch the graph in a session. The graph does not compute anything or hold values; it just defines operations. A session allocates resources and holds values for variables.
A session is launched using session = tf.Session(). Because a session must be closed with session.close(), it is common to use a with statement. You can run an operation in the graph by calling session.run(op_name). This will fetch the output of the operation op_name. If you pass in a list of operation names, calling session.run() will fetch multiple values.
References
A variable maintains state in the graph across all calls to run(). The Variable() constructor requires an initial value, which can be a Tensor of any size and shape. When a session is launched, we need to initialize all variables with an initializer op. This is usually done using tf.global_variables_initializer(), which adds an op to the graph to initialize all variables. Don't forget to run this Op after launching the graph in a session.
import tensorflow as tf
x = tf.constant(35, name='x')
y = tf.Variable(x + 5, name='y')
model = tf.global_variables_initializer()
with tf.Session() as session:
session.run(model)
print(session.run(y))
A placeholder allows you to build computation graphs without needing specific data. We create a placeholder during construction and feed in data when we launch the session. This is done by adding a feed_dict={...} parameter to session.run().
import tensorflow as tf
x = tf.placeholder("float", 3)
y = x * 2
with tf.Session() as session:
result = session.run(y, feed_dict={x: [1, 2, 3]})
print(result)
References
- TensorFlow: Variables
- Learning TensorFlow: Variables
- TensorFlow: Placeholders
- Learning TensorFlow: Placeholders
Let's start with a simple univariate linear model. We want to model house price as a function of the square footage in the form y = Wx + b. Since x is an input, it makes sense to make it a placeholder. W and b are the model parameters that will be learned and reused, so we want to make them variables. During the learning process, we will want to minimize the distance between the predicted output, y, and the actual output, which we will call y_.
# Input to model
x = tf.placeholder(tf.float32)
# Actual house prices
y_ = tf.placeholder(tf.float32)
# Model Parameters
W = tf.Variable(tf.zeros([1, 1]))
b = tf.Variable(tf.zeros([1]))
# Model output
y = tf.matmul(x, W) + b
We can then define the operations to compute tha output and to compute the cost function. Note that tf.matmul() only works on matrices, AKA rank 2 tensors. Because of this we give W the shape [1, 1]. This still makes it a single number, but it is technically a rank 2 matrix that we can use with matmul.
If we ran the graph and fetched the value of y after the matmul op, we would have a rank 2 tensor with shape [1, 1]. We want to reduce this to a scalar, so we use tf.reduce_sum(). Since we do not specify the axis on which reduce_sum should reduce the dimension, all dimensions are reduced, making cost a rank 0 tensor (a scalar).
# Model Parameters
W = tf.Variable(tf.zeros([1, 1]))
b = tf.Variable(tf.zeros([1]))
# Model output
y = tf.matmul(x, W) + b
# Cost function
cost = tf.reduce_sum(tf.pow((y_ - y), 2))
Next, we can use gradient descent to minimize cost. We can do this using the tf.train.GradientDescentOptimizer class, a subclass of tf.train.Optimizer. The tf.train.Optimizer.minimize() function combines calls to the compute_gradients() and apply_gradients() methods.
# Gradient descent
optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01)
train_step = optimizer.minimize(cost)
References
- Gentlest Introduction to TensorFlow #1
- TensorFlow Exception with Matmul
- TensorFlow: Matmul
- TensorFlow: Reduction
- TensorFlow: Optimizers
Now that we understand tensors, sessions, variables, placeholders, and basic TensorFlow syntax, we can make a simple neural network to classfy MNIST digits For a thorough introduction to neural networks through MNIST, see Neural Networks and Deep Learning. Otherwise, just follow along with the TensorFlow: MNIST for ML Beginners
import tensorflow as tf
# Import MNIST data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
# Input images
x = tf.placeholder(tf.float32, [None, 784])
# Model parameters
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
# Model output
y = tf.nn.softmax(tf.matmul(x, W) + b)
# Actual output
y_ = tf.placeholder(tf.float32, [None, 10])
# Cost function
cost = tf.reduce_mean(tf.reduce_sum(tf.pow(y_ - y, 2), axis=1))
training_step = tf.train.GradientDescentOptimizer(0.5).minimize(cost)
# Validate model
accuracy = tf.reduce_mean(tf.cast(tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1)), tf.float32))
# Initialize variables
init = tf.global_variables_initializer()
# Launch session
with tf.Session() as sess:
sess.run(init)
# Train model
for i in range(2000):
batch = mnist.train.next_batch(100)
sess.run(training_step, feed_dict={x: batch[0], y_: batch[1]})
print("Accuracy: %f" % sess.run(
accuracy, feed_dict={x: mnist.test.images, y_:mnist.test.labels}))
References
tf.summary.scalar('name', variable)
...
merged = tf.summary.merge_all()
with tf.Session as sess:
for i = 1 to num_epochs:
writer = tf.summary.FileWriter('/logdir', sess.graph)
summary = sess.run(merged)
writer.add_summary(summary, i)
saver = tf.train.Saver()
saver.save(sess, './model')
...
with tf.Session as sess:
saver.restore(sess, './model')
...
...
tensorboard --logdir=logdir
TensorBoard: Visualizing Learning TensorBoard: Embedding Visualization
TensorFlow programs are build around a tf.Graph object, which contains a set of tf.Operation objects and tf.Tensor objects. The operations represent computation while Tensors represent data. TensorFlow functions add operations and tensors to the default graph. To control this, we can call graphObj.as_default() which returns a context manager to make graphObj the default.
A graph instance supports many collections. Graph collections keep track of data about a graph. We can access the graph collections by calling tf.get_collection(key, scope). The keys for graph collections can be found in tf.GraphKeys, including tf.GraphKeys.TRAINABLE_VARIABLES, tf.GraphKeys.SUMMARIES, etc.
When we make a call to tf.get_collection(), we pass in one of the keys to tell Tensorflow what keys to get from the current default graph. The scope parameter is used to filter results in the collection using regular expressions. This means scope should be as specific as possible.
For example, if we want all of the trainable variables in the Net/N/Layer_1 variable scope, we could use tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, "Net/N"). Note that we can NOT just use "N" as the scope parameter, because "Net/_" matches the regular expression "N".
Note that tf.Graph is not thread safe, and we should construct graphs in a single thread.
References