Notebook following the official Tensorflow tutorials on Tensors, Variables, and Sessions.

tf_tensors_vars_session.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "tf-tensors-vars-session.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ytS65Ti1SGo6",
        "colab_type": "text"
      },
      "source": [
        "This notebook follows the low-level API tutorials on Tensorflow, found [here](https://www.tensorflow.org/guide/tensors), [here](https://www.tensorflow.org/guide/variables), and some of [here](https://www.tensorflow.org/guide/graphs)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gB_TaaW3JoN0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "\n",
        "import numpy as np\n",
        "import tensorflow as tf"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q7LYPsj-KJmO",
        "colab_type": "text"
      },
      "source": [
        "# Tensors\n",
        "\n",
        "Each Tensor has a type and a shape; the type must always be known.   \n",
        "Other Tensors include `tf.Variable, tf.constant, tf.placeholder, tf.SparseTensor`. All but `tf.Variable` are immutable.  \n",
        "You can identify rank and shape of a Tensor."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qLszT7ATG8t6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "62c3a7da-4ca0-4c1d-eb53-9201e3735993"
      },
      "source": [
        "mat = tf.Variable([ [4, 9], [16, 25] ], tf.int32)\n",
        "rank = tf.rank(mat)\n",
        "shape = tf.shape(mat)\n",
        "\n",
        "sess = tf.Session()\n",
        "init = tf.global_variables_initializer()\n",
        "sess.run(init)\n",
        "print(sess.run(rank))\n",
        "print(sess.run(shape))\n",
        "sess.close()"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2\n",
            "[2 2]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pwtBqxFqQriA",
        "colab_type": "text"
      },
      "source": [
        "## Slicing\n",
        "\n",
        "N-dimensional tensors can be accessed element-wise by specifying n indices. They can also be sliced, returning subvectors, submatrices, and other subtensors:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sYFhNe3TJsnc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "fb13b3c2-55f6-4155-ba37-a14eef9c5239"
      },
      "source": [
        "row_v = mat[1]\n",
        "col_v = mat[:,1]\n",
        "\n",
        "sess = tf.Session()\n",
        "init = tf.global_variables_initializer()\n",
        "sess.run(init)\n",
        "print(sess.run(row_v))\n",
        "print(sess.run(col_v))\n",
        "sess.close()"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[16 25]\n",
            "[ 9 25]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GmXmcyA6Rfr6",
        "colab_type": "text"
      },
      "source": [
        "## Reshaping\n",
        "\n",
        "The number of elements in a Tensor is the product of the sizes of its dimensions.  \n",
        "To change the shape of a Tensor while keeping the elements fixed, use `reshape()`. Handily, passing `-1` as an index tells the function to calculate the size of that dimension. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xTJavDMIR5ms",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "rank_three_tensor = tf.ones([3, 4, 5])\n",
        "matrix = tf.reshape(rank_three_tensor, [6, 10])  # Reshape existing content into\n",
        "                                                 # a 6x10 matrix\n",
        "matrixB = tf.reshape(matrix, [3, -1])  #  Reshape existing content into a 3x20\n",
        "                                       # matrix. -1 tells reshape to calculate\n",
        "                                       # the size of this dimension.\n",
        "matrixAlt = tf.reshape(matrixB, [4, 3, -1])  # Reshape existing content into a\n",
        "                                             #4x3x5 tensor\n",
        "\n",
        "# Note that the number of elements of the reshaped Tensors has to match the\n",
        "# original number of elements. Therefore, the following example generates an\n",
        "# error because no possible value for the last dimension will match the number\n",
        "# of elements.\n",
        "#yet_another = tf.reshape(matrixAlt, [13, 2, -1])  # ERROR!"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N-xeIaHzTMI8",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 272
        },
        "outputId": "292c0c36-cec9-446d-e63e-c718f0066aa6"
      },
      "source": [
        "sess = tf.Session()\n",
        "print(sess.run(matrixAlt))\n",
        "sess.close()"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[[1. 1. 1. 1. 1.]\n",
            "  [1. 1. 1. 1. 1.]\n",
            "  [1. 1. 1. 1. 1.]]\n",
            "\n",
            " [[1. 1. 1. 1. 1.]\n",
            "  [1. 1. 1. 1. 1.]\n",
            "  [1. 1. 1. 1. 1.]]\n",
            "\n",
            " [[1. 1. 1. 1. 1.]\n",
            "  [1. 1. 1. 1. 1.]\n",
            "  [1. 1. 1. 1. 1.]]\n",
            "\n",
            " [[1. 1. 1. 1. 1.]\n",
            "  [1. 1. 1. 1. 1.]\n",
            "  [1. 1. 1. 1. 1.]]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ajjmcYsoVQYF",
        "colab_type": "text"
      },
      "source": [
        "## Evaluating tensors\n",
        "\n",
        "Once the graph is build, one can evaluate a Tensor by running the corresponding computation with the function `eval()`. This function needs an active `session`. Its usage is similar to running `sess.run(tensor, feed_dict={})` but it is called on the Tensor of interest.  \n",
        "Note that it is possible to use `feed_dict` on all Tensors not just placeholders."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nH_gtIliUbNN",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "54fe6535-0541-40e0-e1de-fcd0a0a6ace5"
      },
      "source": [
        "constant = tf.constant([1., 2., 3.])\n",
        "x = tf.placeholder(tf.float32)\n",
        "tensor = (constant * constant) + x\n",
        "sess = tf.Session()\n",
        "print(tensor.eval(session=sess, feed_dict={x:[1., 1., 1.]}))\n",
        "sess.close()"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[ 2.  5. 10.]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m8B_zrEnY68J",
        "colab_type": "text"
      },
      "source": [
        "# Variables\n",
        "\n",
        "Variables represent tensors whose values can be changed by running operations. Unlike `tf.Tensor` objects, `tf.Variable` exists outside of a single `session.run()` call.  \n",
        "Inside, it stores a persistent tensor which can be read or modified and these modifications are visible across multiple `tf.Session`s.  \n",
        "\n",
        "Create a variable, which will be randomly initialised by default:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wQSjXQP6Zu0G",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "ac226ddb-9c40-42f6-b48d-cbb89bd3cd8d"
      },
      "source": [
        "my_variable = tf.get_variable(\"my_variable\", [1, 2, 3])"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING: Logging before flag parsing goes to stderr.\n",
            "W0620 16:44:56.050926 139954622445440 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Call initializer instance with the dtype argument instead of passing it to the constructor\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Red1gVWQafRd",
        "colab_type": "text"
      },
      "source": [
        "Or pass an initialiser and specify shape or use an existing Tensor to initialise:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8FF3mPrPaSYQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "my_int_variable = tf.get_variable(\"my_int_variable\", [1, 2, 3], dtype=tf.int32,\n",
        "  initializer=tf.zeros_initializer)\n",
        "other_variable = tf.get_variable(\"other_variable\", dtype=tf.int32,\n",
        "  initializer=tf.constant([23, 42]))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8WkN_GD0a-jt",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "e51ae35b-05c9-471a-85b9-37dcf81ffa5a"
      },
      "source": [
        "init = tf.global_variables_initializer() # initialise all global variables\n",
        "sess = tf.Session()\n",
        "sess.run(init)\n",
        "print(sess.run(my_variable))\n",
        "sess.close()"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[[ 0.12341881 -0.68054676  0.5159533 ]\n",
            "  [ 0.45553112 -0.51104665 -0.6310594 ]]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BBRtQ0NUcUqa",
        "colab_type": "text"
      },
      "source": [
        "## Variable collections\n",
        "\n",
        "In order to access many variables at the same time, Tensorflow has *collections*. By default every variable gets placed in \n",
        "- `tf.GraphKeys.GLOBAL_VARIABLES` --- variables that can be shared across multiple devices,\n",
        "- `tf.GraphKeys.TRAINABLE_VARIABLES` --- variables for which TensorFlow will calculate gradients.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ABiRyjUlc-hO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "3964952f-800c-48fc-ef79-0f88695262bd"
      },
      "source": [
        "my_local = tf.get_variable(\"my_local\", shape=(),\n",
        "collections=[tf.GraphKeys.LOCAL_VARIABLES]) # Add variable to local collection\n",
        "\n",
        "my_non_trainable = tf.get_variable(\"my_non_trainable\",\n",
        "                                   shape=(),\n",
        "                                   trainable=False) # Make a variable non-trainable\n",
        "\n",
        "tf.add_to_collection(\"my_collection_name\", my_local) # Define your own collection\n",
        "tf.get_collection(\"my_collection_name\") # Retrieve all variables in it\n",
        "\n"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<tf.Variable 'my_local:0' shape=() dtype=float32_ref>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fFtxt4ndd95U",
        "colab_type": "text"
      },
      "source": [
        "## Initialising\n",
        "\n",
        "Initialise individual variables or find out which variables are not initialised:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "reZxKmzSeKXl",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "0154edef-473b-4dba-c3be-f4cc8179b86e"
      },
      "source": [
        "sess = tf.Session()\n",
        "sess.run(my_variable.initializer)\n",
        "\n",
        "print(sess.run(tf.report_uninitialized_variables()))\n",
        "sess.close()"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "W0620 16:45:46.755725 139954622445440 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/array_ops.py:1354: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "[b'Variable']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5B2dvW5GebaC",
        "colab_type": "text"
      },
      "source": [
        "Note that `tf.global_variables_initializer()` doesn't specify the order of initialisation so if you ever refer to another variable, it is best to use its initialised value:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NtueypmBelOX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 88
        },
        "outputId": "60c8bc78-efb6-40dd-d37c-79ab57df3f83"
      },
      "source": [
        "v = tf.get_variable(\"v\", shape=(), initializer=tf.zeros_initializer())\n",
        "w = tf.get_variable(\"w\", initializer=v.initialized_value() + 1)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "W0620 16:45:54.215157 139954622445440 deprecation.py:323] From <ipython-input-12-53c9cc7aeb9c>:2: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BCtQky2Gex1l",
        "colab_type": "text"
      },
      "source": [
        "Apparently a lot of these functions are deprecated?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X-m8guHPfEKu",
        "colab_type": "text"
      },
      "source": [
        "## Assigning \n",
        "\n",
        "To assign a value to a variable, use `assign()` and similar functions. Check the value of a variable following some operation with `read_value()`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kV9iuIEhgE1C",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "c0b9241b-aa29-4304-8086-2aeebb2630bd"
      },
      "source": [
        "assignment = v.assign_add(1)\n",
        "sess = tf.Session()\n",
        "tf.global_variables_initializer().run(session=sess)\n",
        "sess.run(assignment)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FiL9HOqTiPw-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "with tf.control_dependencies([assignment]):\n",
        "  w = v.read_value()\n",
        "\n",
        "sess.close()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hsOzmSCvlQkQ",
        "colab_type": "text"
      },
      "source": [
        "## Variable scope\n",
        "\n",
        "Scope allows to call the same function to create e.g. sequential layers (otherwise the ambiguity of creating new variables or reusing old ones makes Tensorflow fail). To re-use variables, one can use `reuse=True` flag or call such property on the scope variable."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1Vwc1P2dl37S",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def conv_relu(input, kernel_shape, bias_shape):\n",
        "    # Create variable named \"weights\".\n",
        "    weights = tf.get_variable(\"weights\", kernel_shape,\n",
        "        initializer=tf.random_normal_initializer())\n",
        "    # Create variable named \"biases\".\n",
        "    biases = tf.get_variable(\"biases\", bias_shape,\n",
        "        initializer=tf.constant_initializer(0.0))\n",
        "    conv = tf.nn.conv2d(input, weights,\n",
        "        strides=[1, 1, 1, 1], padding='SAME')\n",
        "    return tf.nn.relu(conv + biases)\n",
        "  \n",
        "def my_image_filter(input_images):\n",
        "    with tf.variable_scope(\"conv1\"):\n",
        "        # Variables created here will be named \"conv1/weights\", \"conv1/biases\".\n",
        "        relu1 = conv_relu(input_images, [5, 5, 32, 32], [32])\n",
        "    with tf.variable_scope(\"conv2\"):\n",
        "        # Variables created here will be named \"conv2/weights\", \"conv2/biases\".\n",
        "        return conv_relu(relu1, [5, 5, 32, 32], [32])\n",
        "      \n",
        "input1 = tf.random_normal([1,10,10,32])\n",
        "input2 = tf.random_normal([1,20,20,32])\n",
        "      \n",
        "with tf.variable_scope(\"model\"):\n",
        "  output1 = my_image_filter(input1)\n",
        "with tf.variable_scope(\"model\", reuse=True):\n",
        "  output2 = my_image_filter(input2)\n",
        "  \n",
        "#with tf.variable_scope(\"model\") as scope:\n",
        "#  output1 = my_image_filter(input1)\n",
        "#  scope.reuse_variables()\n",
        "#  output2 = my_image_filter(input2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5ENdIShNzxlh",
        "colab_type": "text"
      },
      "source": [
        "# Session\n",
        "\n",
        "`tf.Session` connects the computational graph with the local and remote devices and caches information about the graph for efficiency.\n",
        "\n",
        "Since a `tf.Session` owns physical resources, it is typically used in a `with` block that automatically closes the session when you exit. It is also possible to create a session manually, but it should be explicitly closed when you are finished with it to free the resources."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nYzGxqZQ0wkC",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "3e9efae9-c304-4857-b6de-acdac2be923c"
      },
      "source": [
        "x = tf.constant(1)\n",
        "y = x + 5\n",
        "# Create a default in-process session.\n",
        "with tf.Session() as sess:\n",
        "  print(sess.run(y))"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "6\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dKDP159k2BNF",
        "colab_type": "text"
      },
      "source": [
        "`tf.Session` accepts additional arguments specifying the desired session and its `run()` function allows specifying options about the call and collecting metadata about the execution. More details in [here](https://www.tensorflow.org/guide/graphs#executing_a_graph_in_a_tfsession)."
      ]
    }
  ]
}