Skip to content

Instantly share code, notes, and snippets.

@jarrensj

jarrensj/sequence-to-sequence-learning-with-tensor2tensor.ipynb

Created November 11, 2018 17:09
Show Gist options
  • Save jarrensj/55a0452a57b77eaeb0e7814953563e74 to your computer and use it in GitHub Desktop.
Save jarrensj/55a0452a57b77eaeb0e7814953563e74 to your computer and use it in GitHub Desktop.
Download ZIP
Sequence to Sequence Learning with Tensor2Tensor
Display the source blob
Display the rendered blob
Raw
sequence-to-sequence-learning-with-tensor2tensor.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "Sequence to Sequence Learning with Tensor2Tensor",
"version": "0.3.2",
"provenance": [],
"collapsed_sections": [],
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"accelerator": "GPU"
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/jarrensj/55a0452a57b77eaeb0e7814953563e74/sequence-to-sequence-learning-with-tensor2tensor.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"metadata": {
"id": "odi2vIMHC3Rm",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# Sequence to Sequence Learning with Tensor2Tensor\n"
]
},
{
"metadata": {
"id": "ir4JzdPlsD9M",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"This notebook was created for PhD Open lectures given in June 2018 at the University of Warsaw. The slides for the lectures are available [here](http://phdopen.mimuw.edu.pl/lato18/w5s/DeepLearning.pdf) and the full recording [here on YouTube](https://www.youtube.com/channel/UCvMN-HLvvVa6lUXijCKFlqQ) - take a look at it if you need more explanation!"
]
},
{
"metadata": {
"id": "OPGni6fuvoTj",
"colab_type": "code",
"colab": {}
},
"cell_type": "code",
"source": [
"# Install deps.\n",
"!pip install -q -U tensor2tensor\n",
"!pip install -q tensorflow matplotlib\n"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "xN32gDt-Lj2T",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 2305
},
"outputId": "5c69fe8b-263f-4d1c-c9f9-50753db35e15"
},
"cell_type": "code",
"source": [
"!t2t-trainer --generate_data \\\n",
" --data_dir=~/t2t_data \\\n",
" --output_dir=~/t2t_train/mnist \\\n",
" --problem=image_mnist \\\n",
" --model=shake_shake \\\n",
" --hparams_set=shake_shake_quick \\\n",
" --train_steps=1000 \\\n",
" --eval_steps=100"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
":::MLPv0.5.0 transformer 1541955887.537092447 (/usr/local/bin/t2t-trainer:28) run_start\n",
":::MLPv0.5.0 transformer 1541955887.537731647 (/usr/local/bin/t2t-trainer:28) run_set_random_seed\n",
"INFO:tensorflow:Generating data for image_mnist\n",
"INFO:tensorflow:Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to /tmp/t2t_datagen/train-images-idx3-ubyte.gz\n",
"100% completed\n",
"INFO:tensorflow:Successfully downloaded train-images-idx3-ubyte.gz, 9912422 bytes.\n",
"INFO:tensorflow:Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to /tmp/t2t_datagen/train-labels-idx1-ubyte.gz\n",
"113% completed\n",
"INFO:tensorflow:Successfully downloaded train-labels-idx1-ubyte.gz, 28881 bytes.\n",
"INFO:tensorflow:Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to /tmp/t2t_datagen/t10k-images-idx3-ubyte.gz\n",
"100% completed\n",
"INFO:tensorflow:Successfully downloaded t10k-images-idx3-ubyte.gz, 1648877 bytes.\n",
"INFO:tensorflow:Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to /tmp/t2t_datagen/t10k-labels-idx1-ubyte.gz\n",
"180% completed\n",
"INFO:tensorflow:Successfully downloaded t10k-labels-idx1-ubyte.gz, 4542 bytes.\n",
"INFO:tensorflow:Not downloading, file already found: /tmp/t2t_datagen/train-images-idx3-ubyte.gz\n",
"INFO:tensorflow:Not downloading, file already found: /tmp/t2t_datagen/train-labels-idx1-ubyte.gz\n",
"INFO:tensorflow:Not downloading, file already found: /tmp/t2t_datagen/t10k-images-idx3-ubyte.gz\n",
"INFO:tensorflow:Not downloading, file already found: /tmp/t2t_datagen/t10k-labels-idx1-ubyte.gz\n",
"2018-11-11 17:04:53.236611: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2018-11-11 17:04:53.237034: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: \n",
"name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235\n",
"pciBusID: 0000:00:04.0\n",
"totalMemory: 11.17GiB freeMemory: 11.10GiB\n",
"2018-11-11 17:04:53.237072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0\n",
"2018-11-11 17:04:54.225996: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2018-11-11 17:04:54.226072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 \n",
"2018-11-11 17:04:54.226094: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N \n",
"2018-11-11 17:04:54.226373: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
"2018-11-11 17:04:54.226471: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10758 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
"INFO:tensorflow:Generating case 0.\n",
"INFO:tensorflow:Generated 60000 Examples\n",
"2018-11-11 17:05:26.220023: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0\n",
"2018-11-11 17:05:26.220111: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2018-11-11 17:05:26.220136: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 \n",
"2018-11-11 17:05:26.220154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N \n",
"2018-11-11 17:05:26.220405: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10758 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
"INFO:tensorflow:Generating case 0.\n",
"INFO:tensorflow:Generated 10000 Examples\n",
"INFO:tensorflow:Shuffling data...\n",
"INFO:tensorflow:Data shuffled.\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/trainer_lib.py:230: RunConfig.__init__ (from tensorflow.contrib.learn.python.learn.estimators.run_config) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"When switching to tf.estimator.Estimator, use tf.estimator.RunConfig instead.\n",
"INFO:tensorflow:Configuring DataParallelism to replicate the model.\n",
"INFO:tensorflow:schedule=continuous_train_and_eval\n",
"INFO:tensorflow:worker_gpu=1\n",
"INFO:tensorflow:sync=False\n",
"WARNING:tensorflow:Schedule=continuous_train_and_eval. Assuming that training is running on a single machine.\n",
"INFO:tensorflow:datashard_devices: ['gpu:0']\n",
"INFO:tensorflow:caching_devices: None\n",
"INFO:tensorflow:ps_devices: ['gpu:0']\n",
"INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f758ded5f28>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_train_distribute': None, '_eval_distribute': None, '_device_fn': None, '_tf_config': gpu_options {\n",
" per_process_gpu_memory_fraction: 1.0\n",
"}\n",
", '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_secs': None, '_log_step_count_steps': 100, '_protocol': None, '_session_config': gpu_options {\n",
" per_process_gpu_memory_fraction: 0.95\n",
"}\n",
"allow_soft_placement: true\n",
"graph_options {\n",
" optimizer_options {\n",
" }\n",
"}\n",
", '_save_checkpoints_steps': 1000, '_keep_checkpoint_max': 20, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': '/root/t2t_train/mnist', 'use_tpu': False, 't2t_device_info': {'num_async_replicas': 1}, 'data_parallelism': <tensor2tensor.utils.expert_utils.Parallelism object at 0x7f758ded5978>}\n",
"WARNING:tensorflow:Estimator's model_fn (<function T2TModel.make_estimator_model_fn.<locals>.wrapping_model_fn at 0x7f758df616a8>) includes params argument, but params are not passed to Estimator.\n",
"WARNING:tensorflow:ValidationMonitor only works with --schedule=train_and_evaluate\n",
"INFO:tensorflow:Not using Distribute Coordinator.\n",
"INFO:tensorflow:Running training and evaluation locally (non-distributed).\n",
"INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 1000 or save_checkpoints_secs None.\n",
"INFO:tensorflow:Reading data files from /root/t2t_data/image_mnist-train*\n",
"INFO:tensorflow:partition: 0 num_data_files: 10\n",
":::MLPv0.5.0 transformer 1541955932.138631821 (/usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/problem.py:869) input_order\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/problem.py:653: parallel_interleave (from tensorflow.contrib.data.python.ops.interleave_ops) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Use `tf.data.experimental.parallel_interleave(...)`.\n",
"INFO:tensorflow:Calling model_fn.\n",
"INFO:tensorflow:Setting T2TModel mode to 'train'\n",
":::MLPv0.5.0 transformer 1541955932.482486248 (/usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/t2t_model.py:213) model_hp_initializer_gain: 1.0\n",
"INFO:tensorflow:Using variable initializer: uniform_unit_scaling\n",
"INFO:tensorflow:Transforming feature 'inputs' with image_modality.bottom\n",
"INFO:tensorflow:Transforming feature 'targets' with class_label_modality_10_32.targets_bottom\n",
"INFO:tensorflow:Building model body\n",
"INFO:tensorflow:Transforming body output with class_label_modality_10_32.top\n",
"INFO:tensorflow:Applying exp learning rate warmup for 100 steps\n",
"INFO:tensorflow:Applying learning rate decay: cosine.\n",
"INFO:tensorflow:Base learning rate: 0.500000\n",
"INFO:tensorflow:Applying weight decay, decay_rate: 0.00010\n",
"INFO:tensorflow:Trainable Variables Total size: 2926698\n",
"INFO:tensorflow:Using optimizer Adam\n",
":::MLPv0.5.0 transformer 1541955937.196300745 (/usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/optimize.py:45) opt_name: \"Adam\"\n",
":::MLPv0.5.0 transformer 1541955937.197404385 (/usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/optimize.py:45) opt_hp_Adam_beta1: 0.85\n",
":::MLPv0.5.0 transformer 1541955937.198491573 (/usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/optimize.py:45) opt_hp_Adam_beta2: 0.997\n",
":::MLPv0.5.0 transformer 1541955937.199587822 (/usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/optimize.py:45) opt_hp_Adam_epsilon: 1e-06\n",
"INFO:tensorflow:Done calling model_fn.\n",
"INFO:tensorflow:Create CheckpointSaverHook.\n",
"INFO:tensorflow:Graph was finalized.\n",
"2018-11-11 17:05:47.514159: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0\n",
"2018-11-11 17:05:47.514257: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2018-11-11 17:05:47.514280: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 \n",
"2018-11-11 17:05:47.514311: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N \n",
"2018-11-11 17:05:47.514735: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10758 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
"INFO:tensorflow:Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"INFO:tensorflow:Saving checkpoints for 0 into /root/t2t_train/mnist/model.ckpt.\n",
"INFO:tensorflow:loss = 9.014041, step = 0\n",
"INFO:tensorflow:global_step/sec: 3.84067\n",
"INFO:tensorflow:loss = 1.019277, step = 100 (26.038 sec)\n",
"INFO:tensorflow:global_step/sec: 4.26303\n",
"INFO:tensorflow:loss = 0.45821336, step = 200 (23.457 sec)\n",
"INFO:tensorflow:global_step/sec: 4.26401\n",
"INFO:tensorflow:loss = 0.23634583, step = 300 (23.452 sec)\n",
"INFO:tensorflow:global_step/sec: 4.25292\n",
"INFO:tensorflow:loss = 0.20437743, step = 400 (23.513 sec)\n",
"INFO:tensorflow:global_step/sec: 4.2582\n",
"INFO:tensorflow:loss = 0.07296789, step = 500 (23.484 sec)\n",
"INFO:tensorflow:global_step/sec: 4.2606\n",
"INFO:tensorflow:loss = 0.3940377, step = 600 (23.471 sec)\n",
"INFO:tensorflow:global_step/sec: 4.24909\n",
"INFO:tensorflow:loss = 0.022622515, step = 700 (23.535 sec)\n",
"INFO:tensorflow:global_step/sec: 4.25653\n",
"INFO:tensorflow:loss = 0.13142672, step = 800 (23.493 sec)\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "oILRLCWN_16u",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 672
},
"outputId": "70f2dbd9-4d07-4364-f504-1b6ffe54c8f9"
},
"cell_type": "code",
"source": [
"# Imports we need.\n",
"import tensorflow as tf\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import os\n",
"import collections\n",
"import random\n",
"import six\n",
"\n",
"from tensor2tensor import models\n",
"from tensor2tensor import problems\n",
"from tensor2tensor.layers import common_attention\n",
"from tensor2tensor.layers import common_layers\n",
"from tensor2tensor.utils import trainer_lib\n",
"from tensor2tensor.utils import t2t_model\n",
"from tensor2tensor.utils import registry\n",
"from tensor2tensor.utils import metrics\n",
"from tensor2tensor.utils import learning_rate\n",
"from tensor2tensor.utils import optimize\n",
"\n",
"# TF session\n",
"sess = tf.Session()\n",
"\n",
"# Other setup\n",
"Modes = tf.estimator.ModeKeys\n",
"\n",
"# Setup some directories\n",
"data_dir = os.path.expanduser(\"~/t2t/data\")\n",
"tmp_dir = os.path.expanduser(\"~/t2t/tmp\")\n",
"train_dir = os.path.expanduser(\"~/t2t/train\")\n",
"checkpoint_dir = os.path.expanduser(\"~/t2t/checkpoints\")\n",
"tf.gfile.MakeDirs(data_dir)\n",
"tf.gfile.MakeDirs(tmp_dir)\n",
"tf.gfile.MakeDirs(train_dir)\n",
"tf.gfile.MakeDirs(checkpoint_dir)\n",
"gs_data_dir = \"gs://tensor2tensor-data\"\n",
"gs_ckpt_dir = \"gs://tensor2tensor-checkpoints/\""
],
"execution_count": 3,
"outputs": [
{
"output_type": "stream",
"text": [
"/usr/local/lib/python2.7/dist-packages/tensor2tensor/data_generators/video_generated.py:34: UserWarning: \n",
"This call to matplotlib.use() has no effect because the backend has already\n",
"been chosen; matplotlib.use() must be called *before* pylab, matplotlib.pyplot,\n",
"or matplotlib.backends is imported for the first time.\n",
"\n",
"The backend was *originally* set to 'module://ipykernel.pylab.backend_inline' by the following code:\n",
" File \"/usr/lib/python2.7/runpy.py\", line 174, in _run_module_as_main\n",
" \"__main__\", fname, loader, pkg_name)\n",
" File \"/usr/lib/python2.7/runpy.py\", line 72, in _run_code\n",
" exec code in run_globals\n",
" File \"/usr/local/lib/python2.7/dist-packages/ipykernel_launcher.py\", line 16, in <module>\n",
" app.launch_new_instance()\n",
" File \"/usr/local/lib/python2.7/dist-packages/traitlets/config/application.py\", line 657, in launch_instance\n",
" app.initialize(argv)\n",
" File \"<decorator-gen-121>\", line 2, in initialize\n",
" File \"/usr/local/lib/python2.7/dist-packages/traitlets/config/application.py\", line 87, in catch_config_error\n",
" return method(app, *args, **kwargs)\n",
" File \"/usr/local/lib/python2.7/dist-packages/ipykernel/kernelapp.py\", line 462, in initialize\n",
" self.init_gui_pylab()\n",
" File \"/usr/local/lib/python2.7/dist-packages/ipykernel/kernelapp.py\", line 403, in init_gui_pylab\n",
" InteractiveShellApp.init_gui_pylab(self)\n",
" File \"/usr/local/lib/python2.7/dist-packages/IPython/core/shellapp.py\", line 213, in init_gui_pylab\n",
" r = enable(key)\n",
" File \"/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py\", line 2950, in enable_matplotlib\n",
" pt.activate_matplotlib(backend)\n",
" File \"/usr/local/lib/python2.7/dist-packages/IPython/core/pylabtools.py\", line 309, in activate_matplotlib\n",
" matplotlib.pyplot.switch_backend(backend)\n",
" File \"/usr/local/lib/python2.7/dist-packages/matplotlib/pyplot.py\", line 232, in switch_backend\n",
" matplotlib.use(newbackend, warn=False, force=True)\n",
" File \"/usr/local/lib/python2.7/dist-packages/matplotlib/__init__.py\", line 1305, in use\n",
" reload(sys.modules['matplotlib.backends'])\n",
" File \"/usr/local/lib/python2.7/dist-packages/matplotlib/backends/__init__.py\", line 14, in <module>\n",
" line for line in traceback.format_stack()\n",
"\n",
"\n",
" matplotlib.use(\"agg\")\n"
],
"name": "stderr"
}
]
},
{
"metadata": {
"id": "0a69r1KDiZDe",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# Download MNIST and inspect it"
]
},
{
"metadata": {
"id": "JKc2uSk6WX5e",
"colab_type": "code",
"outputId": "89169ad1-78ac-4bfe-c8fd-e5970eb0840d",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 417
}
},
"cell_type": "code",
"source": [
"# Fetch the MNIST problem\n",
"mnist_problem = problems.problem(\"image_mnist\")\n",
"# The generate_data method of a problem will download data and process it into\n",
"# a standard format ready for training and evaluation.\n",
"mnist_problem.generate_data(data_dir, tmp_dir)"
],
"execution_count": 4,
"outputs": [
{
"output_type": "stream",
"text": [
"INFO:tensorflow:Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to /root/t2t/tmp/train-images-idx3-ubyte.gz\n",
"100% completed\n",
"INFO:tensorflow:Successfully downloaded train-images-idx3-ubyte.gz, 9912422 bytes.\n",
"INFO:tensorflow:Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to /root/t2t/tmp/train-labels-idx1-ubyte.gz\n",
"113% completed\n",
"INFO:tensorflow:Successfully downloaded train-labels-idx1-ubyte.gz, 28881 bytes.\n",
"INFO:tensorflow:Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to /root/t2t/tmp/t10k-images-idx3-ubyte.gz\n",
"100% completed\n",
"INFO:tensorflow:Successfully downloaded t10k-images-idx3-ubyte.gz, 1648877 bytes.\n",
"INFO:tensorflow:Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to /root/t2t/tmp/t10k-labels-idx1-ubyte.gz\n",
"180% completed\n",
"INFO:tensorflow:Successfully downloaded t10k-labels-idx1-ubyte.gz, 4542 bytes.\n",
"INFO:tensorflow:Not downloading, file already found: /root/t2t/tmp/train-images-idx3-ubyte.gz\n",
"INFO:tensorflow:Not downloading, file already found: /root/t2t/tmp/train-labels-idx1-ubyte.gz\n",
"INFO:tensorflow:Not downloading, file already found: /root/t2t/tmp/t10k-images-idx3-ubyte.gz\n",
"INFO:tensorflow:Not downloading, file already found: /root/t2t/tmp/t10k-labels-idx1-ubyte.gz\n",
"INFO:tensorflow:Generating case 0.\n",
"INFO:tensorflow:Generated 60000 Examples\n",
"INFO:tensorflow:Generating case 0.\n",
"INFO:tensorflow:Generated 10000 Examples\n",
"INFO:tensorflow:Shuffling data...\n",
"INFO:tensorflow:Data shuffled.\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "VW6HCRANFPYV",
"colab_type": "code",
"outputId": "a076dd39-cba7-46a2-c546-97e61700c0d1",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 420
}
},
"cell_type": "code",
"source": [
"# Now let's see the training MNIST data as Tensors.\n",
"mnist_data = mnist_problem.dataset(Modes.TRAIN, data_dir)\n",
"mnist_example_tensors = mnist_data.make_one_shot_iterator().get_next()\n",
"mnist_example = sess.run(mnist_example_tensors)\n",
"image = mnist_example[\"inputs\"]\n",
"label = mnist_example[\"targets\"]\n",
"\n",
"plt.imshow(image[:, :, 0].astype(np.float32), cmap=plt.get_cmap('gray'))\n",
"print(\"Label: %d\" % label)"
],
"execution_count": 6,
"outputs": [
{
"output_type": "stream",
"text": [
"INFO:tensorflow:Reading data files from /root/t2t/data/image_mnist-train*\n",
"INFO:tensorflow:partition: 0 num_data_files: 10\n",
":::MLPv0.5.0 transformer 1541815572.724098921 (<ipython-input-6-55840b779871>:1) input_order\n",
"Label: 1\n"
],
"name": "stdout"
},
{
"output_type": "display_data",
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAUsAAAFKCAYAAACU6307AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAEl1JREFUeJzt3W9Ilff/x/HX6RylpMLlVAhWG2FM\n0hgtJYtalgQORn+2G8ulDLqRjCIXEc6lbQhZFo1csMxVG5PBAW8FGygtxiLMlmCoMLRuNIlm2qQV\n2eaf87vx4yuzjp63x3POdc7Z8wHeOJ/rus7n/eY6vLyu65zrHJfP5/MJADCtOU4XAACxgLAEAAPC\nEgAMCEsAMCAsAcCAsAQAC18ESPL719nZOeWyWP2Lx57itS96ip2/SPU1HVckPmfpcrn8jvt8vimX\nxap47EmKz77oKXZEqq/p4tAT7JMePXpUt27dksvlUkVFhVauXBnsUwFA1AsqLG/cuKG7d+/K6/Xq\nzp07qqiokNfrDXVtABA1gnqDp7W1VQUFBZKkZcuW6dGjR3ry5ElICwOAaBLUkeXg4KBWrFgx8XjR\nokUaGBjQ/Pnz/a7f2dmprKwsv8sicMk04uKxJyk++6Kn2OF0X0Ffs/y3QE1kZ2dPuV28XYyOx56k\n+OyLnmJHNLzBE9RpeFpamgYHByceP3jwQKmpqcE8FQDEhKDCct26dWpubpYkdXd3Ky0tbcpTcACI\nB0Gdhq9atUorVqzQ+++/L5fLpSNHjoS6LgCIKnwoPcTisScpPvuip9gRs9csAeC/hrAEAAPCEgAM\nCEsAMCAsAcCAsAQAA8ISAAwISwAwICwBwICwBAADwhIADAhLADAgLAHAgLAEAAPCEgAMCEsAMCAs\nAcCAsAQAA8ISAAwISwAwICwBwICwBAADwhIADAhLADAgLAHAgLAEAAPCEgAMCEsAMCAsAcCAsAQA\nA8ISAAwISwAw8DhdAOLfrl27zOt+8803IZ9/9erVUy574403Jj3u6OgI+fyIDxxZAoABYQkABoQl\nABgQlgBgQFgCgAFhCQAGhCUAGBCWAGBAWAKAAXfwIOxmclfO+Ph4yOffsWOHeRl38GAqHFkCgEFQ\nR5ZtbW3av3+/MjIyJEnLly9XZWVlSAsDgGgS9Gl4bm6u6urqQlkLAEQtTsMBwCDosLx9+7ZKS0u1\nc+dOXbt2LZQ1AUDUcfl8Pt9MN+rv71d7e7sKCwvV19enkpIStbS0KDEx0e/6XV1dysrKmnWxAOCU\noMLyee+9956++OILvfLKK/4ncbn8jvt8vimXxap47EmaXV+jo6PmdcPx0aFjx475Ha+srFR1dfWk\nsaqqqpDPH0m8/mY/z1SCOg2/dOmSzp8/L0kaGBjQw4cPlZ6eHlx1ABADgno3fNOmTTp48KB++ukn\njYyM6LPPPpvyFBwA4kFQYTl//nydPXs21LUAQNQKyTXLgJNwzTLmPd/Xp59+at72888/N68bjmuW\nU0lISNDIyMiksVg/Q/qvvP7COc9U+JwlABgQlgBgQFgCgAFhCQAGhCUAGBCWAGBAWAKAAWEJAAaE\nJQAYEJYAYMCvOyIoW7dudboEIKI4sgQAA8ISAAwISwAwICwBwICwBAADwhIADAhLADAgLAHAgLAE\nAAPu4MEk9fX1pmWrVq0yP+ecOdH7P/n52n799VfTdjk5OeEoB1Esel/FABBFCEsAMCAsAcCAsAQA\nA8ISAAwISwAwICwBwICwBAADwhIADAhLADDgdkdMMj4+blo23Xrhmj/U3G73C/P5fL6IzY/YwpEl\nABgQlgBgQFgCgAFhCQAGhCUAGBCWAGBAWAKAAWEJAAaEJQAYEJYAYEBYAoCBKSx7enpUUFCgxsZG\nSdL9+/dVXFysoqIi7d+/X//8809YiwQApwUMy6dPn6q6ulp5eXkTY3V1dSoqKtL333+vpUuXqqmp\nKaxFAoDTAoZlYmKiGhoalJaWNjHW1tamzZs3S5Ly8/PV2toavgoBIAoE/Io2j8cjj2fyasPDw0pM\nTJQkpaSkaGBgIDzVAUCUmPX3WVq+/6+zs1NZWVlBbx9r4rEnSSotLQ37HG63O+xz/FtCQsKkxzk5\nOabtonkfR3Nts+F0X0GFZVJSkp49e6a5c+eqv79/0im6P9nZ2X7HfT6fXC5XMCVErVjv6auvvvI7\nXlpaqrNnz0483r17t/k558yxf+gikl/+m5CQoJGRkUljHR0dpm1zc3PDUdKsxfrrbyqR6mu6QA7q\no0Nr165Vc3OzJKmlpUXr168PrjIAiBEBjyy7urp0/Phx3bt3Tx6PR83NzTp58qTKy8vl9Xq1ePFi\nbdu2LRK1AoBjAoZlVlaWvvvuuxfGL168GJaCACAauXwRuGo61bWGeLy+Eus9LVmyxO/43bt3tXTp\n0onHP/zwg/k5MzMzzetyzXJ2Yv31N5WYvWYJAP81hCUAGBCWAGBAWAKAAWEJAAaEJQAYEJYAYEBY\nAoABYQkABoQlABjM+vssEV9+//130zLrbYGStGLFilnVFE7Pf31cPN4qiNDgyBIADAhLADAgLAHA\ngLAEAAPCEgAMCEsAMCAsAcCAsAQAA8ISAAwISwAw4NcdQywee5Jm19fo6Kh5XX7dcXZ4/c1+nqlw\nZAkABoQlABgQlgBgQFgCgAFhCQAGhCUAGBCWAGBAWAKAAWEJAAaEJQAYEJYAYEBYAoABYQkABoQl\nABgQlgBgQFgCgAFhCQAGhCUAGBCWAGBAWAKAAWEJAAaEJQAYmMKyp6dHBQUFamxslCSVl5frnXfe\nUXFxsYqLi/Xzzz+Hs0YAcJwn0ApPnz5VdXW18vLyJo0fOHBA+fn5YSsMAKJJwCPLxMRENTQ0KC0t\nLRL1AEBUCnhk6fF45PG8uFpjY6MuXryolJQUVVZWatGiRVM+R2dnp7Kysvwu8/l8Myg3NsRjT1Jk\n+nK73WGf498SEhImPc7JyTFtF837OJprmw2n+woYlv5s3bpVycnJyszM1Llz53TmzBlVVVVNuX52\ndrbfcZ/PJ5fLFUwJUSsee5Jm19fo6Kh53fHx8aDmCEZCQoJGRkYmjXV0dJi2zc3NDUdJs8brb/bz\nTCWod8Pz8vKUmZkpSdq0aZN6enqCqwwAYkRQYblv3z719fVJktra2pSRkRHSogAg2gQ8De/q6tLx\n48d17949eTweNTc3a9euXSorK9O8efOUlJSkmpqaSNQKAI5x+SJw1XSqaw3xeH0lHnuSuGb5PK5Z\nRlY0XLMM6g0eYCbmzIneG8Wery0egwahEb2vYgCIIoQlABgQlgBgQFgCgAFhCQAGhCUAGBCWAGBA\nWAKAAWEJAAaEJQAYcLsjgrJ9+3bzujO53zuS94a73e4X5nP6C2YRvTiyBAADwhIADAhLADAgLAHA\ngLAEAAPCEgAMCEsAMCAsAcCAsAQAA+7gQVA++eQTp0sAIoojSwAwICwBwICwBAADwhIADAhLADAg\nLAHAgLAEAAPCEgAMCEsAMCAsAcCA2x0RlJKSEvO6XV1dYawEiAyOLAHAgLAEAAPCEgAMCEsAMCAs\nAcCAsAQAA8ISAAwISwAwICwBwICwBAADbndEUH777Tfzum63O4yVzM6cOXOmfQz8jyksa2tr1d7e\nrtHRUe3Zs0fZ2dk6dOiQxsbGlJqaqhMnTigxMTHctQKAYwKG5fXr19Xb2yuv16uhoSFt375deXl5\nKioqUmFhoU6dOqWmpiYVFRVFol4AcETAc46cnBydPn1akrRw4UINDw+rra1NmzdvliTl5+ertbU1\nvFUCgMMChqXb7VZSUpIkqampSRs2bNDw8PDEaXdKSooGBgbCWyUAOMz8Bs/ly5fV1NSkCxcuaMuW\nLRPjPp8v4LadnZ3Kysryu8yyfayJx56kyPQV6TeDnp/vzTffNG0Xzfs4mmubDaf7MoXl1atXdfbs\nWX399ddasGCBkpKS9OzZM82dO1f9/f1KS0ubdvvs7Gy/4z6fTy6Xa+ZVR7F47EmaXV8zeZGPjY0F\nNUcw3G73C/N1dHSYtl29enU4Spo1Xn+zn2cqAU/DHz9+rNraWtXX1ys5OVmStHbtWjU3N0uSWlpa\ntH79+hCVCgDRKeCR5Y8//qihoSGVlZVNjB07dkyHDx+W1+vV4sWLtW3btrAWCQBOc/kicCFgqsPn\neDxliMeeJE7Dn8dpeGRFw2k4d/Ag7GYSgOPj42GsZDK32/3CfJGcH7GFe7sAwICwBAADwhIADAhL\nADAgLAHAgLAEAAPCEgAMCEsAMCAsAcCAsAQAA8ISAAwISwAwICwBwICwBAADwhIADAhLADAgLAHA\ngLAEAAPCEgAMCEsAMCAsAcCAX3dE2BUXF5vX/fbbb8NYSWAlJSWOzo/oxZElABgQlgBgQFgCgAFh\nCQAGhCUAGBCWAGBAWAKAAWEJAAaEJQAYuHw+ny/sk7hcfsd9Pt+Uy2JVPPYkxWdf9BQ7ItXXdHHI\nkSUAGBCWAGBAWAKAAWEJAAaEJQAYEJYAYEBYAoABYQkABoQlABgQlgBgQFgCgIHp1x1ra2vV3t6u\n0dFR7dmzR1euXFF3d7eSk5MlSbt379bGjRvDWScAOCpgWF6/fl29vb3yer0aGhrS9u3btWbNGh04\ncED5+fmRqBEAHBcwLHNycrRy5UpJ0sKFCzU8PKyxsbGwFwYA0WRGX9Hm9Xp18+ZNud1uDQwMaGRk\nRCkpKaqsrNSiRYumnoSvaIt58dgXPcWOaPiKNnNYXr58WfX19bpw4YK6urqUnJyszMxMnTt3Tn/8\n8Yeqqqqm3Larq0tZWVkzrxwAooXP4JdffvG9++67vqGhoReW9fb2+j744INpt5fk92+6ZbH6F489\nxWtf9BQ7f5HqazoBPzr0+PFj1dbWqr6+fuLd73379qmvr0+S1NbWpoyMjEBPAwAxLeAbPD/++KOG\nhoZUVlY2MbZjxw6VlZVp3rx5SkpKUk1NTViLBACn8Rs8IRaPPUnx2Rc9xY5I9TVdHHIHDwAYEJYA\nYEBYAoABYQkABoQlABgQlgBgQFgCgAFhCQAGhCUAGBCWAGBAWAKAAWEJAAaEJQAYEJYAYEBYAoAB\nYQkABoQlABgQlgBgQFgCgAFhCQAGhCUAGBCWAGAQkZ/CBYBYx5ElABgQlgBgQFgCgAFhCQAGhCUA\nGBCWAGDgcWLSo0eP6tatW3K5XKqoqNDKlSudKCOk2tratH//fmVkZEiSli9frsrKSoerCl5PT48+\n+ugjffjhh9q1a5fu37+vQ4cOaWxsTKmpqTpx4oQSExOdLnNGnu+pvLxc3d3dSk5OliTt3r1bGzdu\ndLbIGaqtrVV7e7tGR0e1Z88eZWdnx/x+kl7s68qVK47vq4iH5Y0bN3T37l15vV7duXNHFRUV8nq9\nkS4jLHJzc1VXV+d0GbP29OlTVVdXKy8vb2Ksrq5ORUVFKiws1KlTp9TU1KSioiIHq5wZfz1J0oED\nB5Sfn+9QVbNz/fp19fb2yuv1amhoSNu3b1deXl5M7yfJf19r1qxxfF9F/DS8tbVVBQUFkqRly5bp\n0aNHevLkSaTLwDQSExPV0NCgtLS0ibG2tjZt3rxZkpSfn6/W1lanyguKv55iXU5Ojk6fPi1JWrhw\noYaHh2N+P0n++xobG3O4KgfCcnBwUC+99NLE40WLFmlgYCDSZYTF7du3VVpaqp07d+ratWtOlxM0\nj8ejuXPnThobHh6eOJ1LSUmJuX3mrydJamxsVElJiT7++GP9+eefDlQWPLfbraSkJElSU1OTNmzY\nEPP7SfLfl9vtdnxfOXLN8t/i5W7LV199VXv37lVhYaH6+vpUUlKilpaWmLxeFEi87LOtW7cqOTlZ\nmZmZOnfunM6cOaOqqiqny5qxy5cvq6mpSRcuXNCWLVsmxmN9P/27r66uLsf3VcSPLNPS0jQ4ODjx\n+MGDB0pNTY10GSGXnp6ut99+Wy6XS0uWLNHLL7+s/v5+p8sKmaSkJD179kyS1N/fHxens3l5ecrM\nzJQkbdq0ST09PQ5XNHNXr17V2bNn1dDQoAULFsTNfnq+r2jYVxEPy3Xr1qm5uVmS1N3drbS0NM2f\nPz/SZYTcpUuXdP78eUnSwMCAHj58qPT0dIerCp21a9dO7LeWlhatX7/e4Ypmb9++ferr65P0/9dk\n//dJhljx+PFj1dbWqr6+fuJd4njYT/76ioZ95ci3Dp08eVI3b96Uy+XSkSNH9Prrr0e6hJB78uSJ\nDh48qL/++ksjIyPau3ev3nrrLafLCkpXV5eOHz+ue/fuyePxKD09XSdPnlR5ebn+/vtvLV68WDU1\nNUpISHC6VDN/Pe3atUvnzp3TvHnzlJSUpJqaGqWkpDhdqpnX69WXX36p1157bWLs2LFjOnz4cMzu\nJ8l/Xzt27FBjY6Oj+4qvaAMAA+7gAQADwhIADAhLADAgLAHAgLAEAAPCEgAMCEsAMCAsAcDg/wBF\nX0YCcbkI8QAAAABJRU5ErkJggg==\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7fc844288210>"
]
},
"metadata": {
"tags": []
}
}
]
},
{
"metadata": {
"id": "gXL7_bVH49Kl",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# Basic MNIST model."
]
},
{
"metadata": {
"id": "jQlCd98qijmO",
"colab_type": "code",
"colab": {}
},
"cell_type": "code",
"source": [
"# One-off all code together.\n",
"\n",
"# Data.\n",
"BATCH_SIZE = 100\n",
"data_batches = mnist_data.repeat().batch(BATCH_SIZE).make_one_shot_iterator()\n",
"batch = data_batches.get_next()\n",
"x, y = batch[\"inputs\"], batch[\"targets\"]\n",
"x = tf.reshape(x, [BATCH_SIZE, 28*28]) # Height and width on channels.\n",
"y = tf.squeeze(y, axis=1) # Bogus dimension.\n",
"# Model.\n",
"hidden_size = 128\n",
"h = tf.layers.dense(x, hidden_size, activation=tf.nn.relu, name=\"hidden\")\n",
"o = tf.layers.dense(h, 10, activation=tf.nn.relu, name=\"output\")\n",
"loss = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=o, labels=y)\n",
"accuracy = tf.to_float(tf.equal(tf.argmax(o, axis=-1), y))\n",
"loss_t, accuracy_t = tf.reduce_mean(loss), tf.reduce_mean(accuracy)\n",
"# Gradients.\n",
"train_op = tf.train.AdamOptimizer().minimize(loss_t)\n",
"sess.run(tf.global_variables_initializer())"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "37NrUjoulXJz",
"colab_type": "code",
"outputId": "49c213a0-3b65-4a51-93c2-acda4d61c8b8",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 235
}
},
"cell_type": "code",
"source": [
"# Train.\n",
"num_steps = 1200 # 2 epochs on 60K examples.\n",
"for step in range(num_steps):\n",
" _, loss, accuracy = sess.run([train_op, loss_t, accuracy_t])\n",
" if step % 100 == 0:\n",
" print(\"Step %d loss %.4f accuracy %.2f\" % (step, loss, accuracy))"
],
"execution_count": 8,
"outputs": [
{
"output_type": "stream",
"text": [
"Step 0 loss 2.8189 accuracy 0.07\n",
"Step 100 loss 0.5726 accuracy 0.81\n",
"Step 200 loss 0.2678 accuracy 0.92\n",
"Step 300 loss 0.2792 accuracy 0.92\n",
"Step 400 loss 0.1361 accuracy 0.97\n",
"Step 500 loss 0.0879 accuracy 0.97\n",
"Step 600 loss 0.1535 accuracy 0.95\n",
"Step 700 loss 0.1934 accuracy 0.96\n",
"Step 800 loss 0.1619 accuracy 0.96\n",
"Step 900 loss 0.1374 accuracy 0.95\n",
"Step 1000 loss 0.1615 accuracy 0.95\n",
"Step 1100 loss 0.0803 accuracy 0.98\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "xilLpmrK6wMB",
"colab_type": "code",
"outputId": "dc0228fe-1ba9-46bc-c0d8-1e8615921d23",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 328
}
},
"cell_type": "code",
"source": [
"# Once again, all refactored and with reset.\n",
"\n",
"# Reset.\n",
"sess.close()\n",
"tf.reset_default_graph()\n",
"sess = tf.Session()\n",
"\n",
"# Data.\n",
"problem = problems.problem(\"image_mnist\")\n",
"problem.generate_data(data_dir, tmp_dir)\n",
"BATCH_SIZE = 100\n",
"train_data = problem.dataset(Modes.TRAIN, data_dir)\n",
"train_batches = train_data.repeat().batch(BATCH_SIZE)\n",
"train_batch = train_batches.make_one_shot_iterator().get_next()\n",
"eval_data = problem.dataset(Modes.EVAL, data_dir)\n",
"eval_batches = eval_data.repeat().batch(BATCH_SIZE)\n",
"eval_batch = eval_batches.make_one_shot_iterator().get_next()\n",
"\n",
"# Model\n",
"def model(batch, mode):\n",
" with tf.variable_scope(\"mymodel\", reuse=mode == Modes.EVAL):\n",
" # Inputs.\n",
" x, y = batch[\"inputs\"], batch[\"targets\"]\n",
" x = tf.reshape(x, [BATCH_SIZE, 28*28]) # Height and width on channels.\n",
" y = tf.squeeze(y, axis=1) # Bogus dimension.\n",
" # Body.\n",
" hidden_size = 128\n",
" h = tf.layers.dense(x, hidden_size, activation=tf.nn.relu, name=\"hidden\")\n",
" o = tf.layers.dense(h, 10, activation=tf.nn.relu, name=\"output\")\n",
" # Loss and accuracy.\n",
" loss = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=o, labels=y)\n",
" accuracy = tf.to_float(tf.equal(tf.argmax(o, axis=-1), y))\n",
" return tf.reduce_mean(loss), tf.reduce_mean(accuracy)\n",
"\n",
"# Model for train.\n",
"train_loss, train_accuracy = model(train_batch, Modes.TRAIN)\n",
"# Gradients.\n",
"train_op = tf.train.AdamOptimizer().minimize(train_loss)\n",
"# Model for eval.\n",
"eval_loss, eval_accuracy = model(eval_batch, Modes.EVAL)"
],
"execution_count": 9,
"outputs": [
{
"output_type": "stream",
"text": [
"INFO:tensorflow:Not downloading, file already found: /root/t2t/tmp/train-images-idx3-ubyte.gz\n",
"INFO:tensorflow:Not downloading, file already found: /root/t2t/tmp/train-labels-idx1-ubyte.gz\n",
"INFO:tensorflow:Not downloading, file already found: /root/t2t/tmp/t10k-images-idx3-ubyte.gz\n",
"INFO:tensorflow:Not downloading, file already found: /root/t2t/tmp/t10k-labels-idx1-ubyte.gz\n",
"INFO:tensorflow:Not downloading, file already found: /root/t2t/tmp/train-images-idx3-ubyte.gz\n",
"INFO:tensorflow:Not downloading, file already found: /root/t2t/tmp/train-labels-idx1-ubyte.gz\n",
"INFO:tensorflow:Not downloading, file already found: /root/t2t/tmp/t10k-images-idx3-ubyte.gz\n",
"INFO:tensorflow:Not downloading, file already found: /root/t2t/tmp/t10k-labels-idx1-ubyte.gz\n",
"INFO:tensorflow:Skipping generator because outputs files exists at ['/root/t2t/data/image_mnist-unshuffled-train-00000-of-00010', '/root/t2t/data/image_mnist-unshuffled-train-00001-of-00010', '/root/t2t/data/image_mnist-unshuffled-train-00002-of-00010', '/root/t2t/data/image_mnist-unshuffled-train-00003-of-00010', '/root/t2t/data/image_mnist-unshuffled-train-00004-of-00010', '/root/t2t/data/image_mnist-unshuffled-train-00005-of-00010', '/root/t2t/data/image_mnist-unshuffled-train-00006-of-00010', '/root/t2t/data/image_mnist-unshuffled-train-00007-of-00010', '/root/t2t/data/image_mnist-unshuffled-train-00008-of-00010', '/root/t2t/data/image_mnist-unshuffled-train-00009-of-00010']\n",
"INFO:tensorflow:Skipping generator because outputs files exists at ['/root/t2t/data/image_mnist-unshuffled-dev-00000-of-00001']\n",
"INFO:tensorflow:Skipping shuffle because output files exist\n",
"INFO:tensorflow:Reading data files from /root/t2t/data/image_mnist-train*\n",
"INFO:tensorflow:partition: 0 num_data_files: 10\n",
":::MLPv0.5.0 transformer 1541815626.600686073 (<ipython-input-9-2915dbeb35ed>:9) input_order\n",
"INFO:tensorflow:Reading data files from /root/t2t/data/image_mnist-dev*\n",
"INFO:tensorflow:partition: 0 num_data_files: 1\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "lYiY9Qz09ZUl",
"colab_type": "code",
"outputId": "01ae888d-f4aa-40a8-a60c-9dd53d4af672",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 926
}
},
"cell_type": "code",
"source": [
"# Train.\n",
"sess.run(tf.global_variables_initializer())\n",
"num_steps = 2400 # 4 epochs on 60K examples.\n",
"for step in range(num_steps + 1):\n",
" _, loss, accuracy = sess.run([train_op, train_loss, train_accuracy])\n",
" if step % 100 == 0:\n",
" print(\"Step %d train loss %.4f accuracy %.2f\" % (step, loss, accuracy))\n",
" loss, accuracy = sess.run([eval_loss, eval_accuracy])\n",
" print(\"Step %d eval loss %.4f accuracy %.2f\" % (step, loss, accuracy))\n"
],
"execution_count": 11,
"outputs": [
{
"output_type": "stream",
"text": [
"Step 0 train loss 2.6255 accuracy 0.10\n",
"Step 0 eval loss 2.2762 accuracy 0.21\n",
"Step 100 train loss 0.3214 accuracy 0.92\n",
"Step 100 eval loss 0.3057 accuracy 0.91\n",
"Step 200 train loss 0.3334 accuracy 0.93\n",
"Step 200 eval loss 0.2190 accuracy 0.95\n",
"Step 300 train loss 0.1749 accuracy 0.93\n",
"Step 300 eval loss 0.1198 accuracy 0.97\n",
"Step 400 train loss 0.0755 accuracy 0.98\n",
"Step 400 eval loss 0.2269 accuracy 0.93\n",
"Step 500 train loss 0.0725 accuracy 0.99\n",
"Step 500 eval loss 0.1052 accuracy 0.97\n",
"Step 600 train loss 0.0861 accuracy 0.98\n",
"Step 600 eval loss 0.0415 accuracy 0.99\n",
"Step 700 train loss 0.1162 accuracy 0.96\n",
"Step 700 eval loss 0.1729 accuracy 0.95\n",
"Step 800 train loss 0.2777 accuracy 0.94\n",
"Step 800 eval loss 0.0766 accuracy 0.97\n",
"Step 900 train loss 0.0839 accuracy 0.98\n",
"Step 900 eval loss 0.0850 accuracy 0.96\n",
"Step 1000 train loss 0.1185 accuracy 0.96\n",
"Step 1000 eval loss 0.1408 accuracy 0.95\n",
"Step 1100 train loss 0.1048 accuracy 0.96\n",
"Step 1100 eval loss 0.0854 accuracy 0.97\n",
"Step 1200 train loss 0.1472 accuracy 0.95\n",
"Step 1200 eval loss 0.1223 accuracy 0.96\n",
"Step 1300 train loss 0.1169 accuracy 0.96\n",
"Step 1300 eval loss 0.0783 accuracy 0.98\n",
"Step 1400 train loss 0.1571 accuracy 0.96\n",
"Step 1400 eval loss 0.0853 accuracy 0.97\n",
"Step 1500 train loss 0.1464 accuracy 0.96\n",
"Step 1500 eval loss 0.0577 accuracy 0.99\n",
"Step 1600 train loss 0.1677 accuracy 0.93\n",
"Step 1600 eval loss 0.1208 accuracy 0.98\n",
"Step 1700 train loss 0.0392 accuracy 0.99\n",
"Step 1700 eval loss 0.0390 accuracy 0.99\n",
"Step 1800 train loss 0.1006 accuracy 0.97\n",
"Step 1800 eval loss 0.0649 accuracy 0.97\n",
"Step 1900 train loss 0.0488 accuracy 0.98\n",
"Step 1900 eval loss 0.1132 accuracy 0.96\n",
"Step 2000 train loss 0.0643 accuracy 0.97\n",
"Step 2000 eval loss 0.0987 accuracy 0.95\n",
"Step 2100 train loss 0.0399 accuracy 0.98\n",
"Step 2100 eval loss 0.1426 accuracy 0.98\n",
"Step 2200 train loss 0.0287 accuracy 0.99\n",
"Step 2200 eval loss 0.0658 accuracy 0.98\n",
"Step 2300 train loss 0.0744 accuracy 0.97\n",
"Step 2300 eval loss 0.0226 accuracy 0.99\n",
"Step 2400 train loss 0.0252 accuracy 0.99\n",
"Step 2400 eval loss 0.1672 accuracy 0.96\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "8XWHdOtcRO7W",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# Simple sequence models"
]
},
{
"metadata": {
"id": "vGbrR07mYFSh",
"colab_type": "code",
"outputId": "df6b6e9d-073e-44ea-f261-d6efbba09b2a",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"cell_type": "code",
"source": [
"def generator(l):\n",
" inputs = list(np.random.randint(2, size=l))\n",
" even = [x for i, x in enumerate(inputs) if i % 2 == 0]\n",
" repeated = [[x, x] for x in even]\n",
" targets = [z for p in repeated for z in p]\n",
" yield {\"inputs\": inputs, \"targets\": targets}\n",
"\n",
"generator1 = lambda: generator(10)\n",
"generator2 = lambda: generator(20)\n",
"\n",
"print(six.next(generator(6)))"
],
"execution_count": 12,
"outputs": [
{
"output_type": "stream",
"text": [
"{'inputs': [0, 0, 1, 0, 1, 1], 'targets': [0, 0, 1, 1, 1, 1]}\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "cBhZ2-I1RXF5",
"colab_type": "code",
"colab": {}
},
"cell_type": "code",
"source": [
"# Sequence model with 1 conv layer.\n",
"\n",
"# Reset.\n",
"sess.close()\n",
"tf.reset_default_graph()\n",
"sess = tf.Session()\n",
"\n",
"# Data.\n",
"types = {\"inputs\": tf.int64, \"targets\": tf.int64}\n",
"shapes = {\"inputs\": tf.TensorShape([None]), \"targets\": tf.TensorShape([None])}\n",
"\n",
"train_data = tf.data.Dataset.from_generator(\n",
" generator1, output_types=types, output_shapes=shapes)\n",
"eval_data = tf.data.Dataset.from_generator(\n",
" generator2, output_types=types, output_shapes=shapes)\n",
"BATCH_SIZE = 100\n",
"train_batches = train_data.repeat().batch(BATCH_SIZE)\n",
"train_batch = train_batches.make_one_shot_iterator().get_next()\n",
"eval_batches = eval_data.repeat().batch(BATCH_SIZE)\n",
"eval_batch = eval_batches.make_one_shot_iterator().get_next()\n",
"\n",
"# Model\n",
"def model(batch, mode):\n",
" with tf.variable_scope(\"mymodel\", reuse=mode == Modes.EVAL):\n",
" # Inputs.\n",
" x, y = batch[\"inputs\"], batch[\"targets\"]\n",
" x = tf.reshape(x, [BATCH_SIZE, -1, 1])\n",
" y = tf.reshape(y, [BATCH_SIZE, -1, 1])\n",
" x_hot = tf.one_hot(x, 2) # From ints to 1-hot vectors.\n",
" y_hot = tf.one_hot(y, 2)\n",
" x = tf.layers.dense(x_hot, 32, name=\"embedding_x\")\n",
" y_emb = tf.layers.dense(y_hot, 32, name=\"embedding_y\")\n",
" # Exercise: try enabling the 2 lines below.\n",
" # positions = tf.get_variable(\"positions\", [1, 20, 1, 32])\n",
" # x += positions[:, :tf.shape(x)[1], :, :]\n",
" # Body.\n",
" hidden_size = 32\n",
" h = tf.layers.conv2d(x, hidden_size, (3, 1),\n",
" padding=\"same\", activation=tf.nn.relu, name=\"hidden\")\n",
" o = tf.layers.conv2d(h, 2, (1, 1),\n",
" padding=\"same\", name=\"output\")\n",
" # Loss and accuracy.\n",
" loss = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=o, labels=y)\n",
" accuracy = tf.to_float(tf.equal(tf.argmax(o, axis=-1), y))\n",
" return tf.reduce_mean(loss), tf.reduce_mean(accuracy)\n",
"\n",
"# Model for train.\n",
"train_loss, train_accuracy = model(train_batch, Modes.TRAIN)\n",
"# Gradients.\n",
"train_op = tf.train.AdamOptimizer().minimize(train_loss)\n",
"# Model for eval.\n",
"eval_loss, eval_accuracy = model(eval_batch, Modes.EVAL)"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "gXT8zEbpXaao",
"colab_type": "code",
"outputId": "b13748ea-a8ff-471d-84dd-99296e5a704d",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 781
}
},
"cell_type": "code",
"source": [
"# Train.\n",
"sess.run(tf.global_variables_initializer())\n",
"num_steps = 200\n",
"for step in range(num_steps + 1):\n",
" _, loss, accuracy = sess.run([train_op, train_loss, train_accuracy])\n",
" if step % 10 == 0:\n",
" print(\"Step %d train loss %.4f accuracy %.2f\" % (step, loss, accuracy))\n",
" loss, accuracy = sess.run([eval_loss, eval_accuracy])\n",
" print(\"Step %d eval loss %.4f accuracy %.2f\" % (step, loss, accuracy))"
],
"execution_count": 14,
"outputs": [
{
"output_type": "stream",
"text": [
"Step 0 train loss 0.6917 accuracy 0.61\n",
"Step 0 eval loss 0.6735 accuracy 0.62\n",
"Step 10 train loss 0.5660 accuracy 0.73\n",
"Step 10 eval loss 0.5456 accuracy 0.75\n",
"Step 20 train loss 0.4568 accuracy 0.76\n",
"Step 20 eval loss 0.4566 accuracy 0.74\n",
"Step 30 train loss 0.4239 accuracy 0.73\n",
"Step 30 eval loss 0.4003 accuracy 0.75\n",
"Step 40 train loss 0.3743 accuracy 0.81\n",
"Step 40 eval loss 0.3639 accuracy 0.78\n",
"Step 50 train loss 0.3254 accuracy 0.81\n",
"Step 50 eval loss 0.3353 accuracy 0.79\n",
"Step 60 train loss 0.3493 accuracy 0.79\n",
"Step 60 eval loss 0.3367 accuracy 0.78\n",
"Step 70 train loss 0.3268 accuracy 0.80\n",
"Step 70 eval loss 0.3382 accuracy 0.78\n",
"Step 80 train loss 0.3181 accuracy 0.80\n",
"Step 80 eval loss 0.3506 accuracy 0.76\n",
"Step 90 train loss 0.3165 accuracy 0.80\n",
"Step 90 eval loss 0.3382 accuracy 0.77\n",
"Step 100 train loss 0.3028 accuracy 0.83\n",
"Step 100 eval loss 0.3270 accuracy 0.79\n",
"Step 110 train loss 0.3093 accuracy 0.80\n",
"Step 110 eval loss 0.3314 accuracy 0.78\n",
"Step 120 train loss 0.3137 accuracy 0.80\n",
"Step 120 eval loss 0.3404 accuracy 0.76\n",
"Step 130 train loss 0.3091 accuracy 0.80\n",
"Step 130 eval loss 0.3298 accuracy 0.78\n",
"Step 140 train loss 0.3130 accuracy 0.81\n",
"Step 140 eval loss 0.3277 accuracy 0.78\n",
"Step 150 train loss 0.3118 accuracy 0.80\n",
"Step 150 eval loss 0.3315 accuracy 0.77\n",
"Step 160 train loss 0.3137 accuracy 0.80\n",
"Step 160 eval loss 0.3307 accuracy 0.77\n",
"Step 170 train loss 0.3057 accuracy 0.79\n",
"Step 170 eval loss 0.3302 accuracy 0.77\n",
"Step 180 train loss 0.2891 accuracy 0.81\n",
"Step 180 eval loss 0.3191 accuracy 0.78\n",
"Step 190 train loss 0.2929 accuracy 0.80\n",
"Step 190 eval loss 0.3340 accuracy 0.77\n",
"Step 200 train loss 0.2974 accuracy 0.80\n",
"Step 200 eval loss 0.3285 accuracy 0.77\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "jfLujWj4jZmc",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# Autoregressive sequence models"
]
},
{
"metadata": {
"id": "Ohk22OBUjiry",
"colab_type": "code",
"outputId": "3c7784eb-eaa8-4c60-c018-c7ac8f197a40",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"cell_type": "code",
"source": [
"def generator(l):\n",
" inputs = list(np.random.randint(2, size=l))\n",
" even = [x for i, x in enumerate(inputs) if i % 2 == 0]\n",
" repeated = [[x, x] for x in even]\n",
" targets1 = [z for p in repeated for z in p]\n",
" targets2 = inputs\n",
" targets = random.choice([targets1, targets2])\n",
" yield {\"inputs\": inputs, \"targets\": targets}\n",
"\n",
"generator1 = lambda: generator(10)\n",
"generator2 = lambda: generator(20)\n",
"\n",
"print(six.next(generator(6)))"
],
"execution_count": 15,
"outputs": [
{
"output_type": "stream",
"text": [
"{'inputs': [0, 0, 1, 0, 0, 0], 'targets': [0, 0, 1, 0, 0, 0]}\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "ZyBn6jzOjS0S",
"colab_type": "code",
"colab": {}
},
"cell_type": "code",
"source": [
"# Sequence model with 1 conv layer.\n",
"\n",
"# Reset.\n",
"sess.close()\n",
"tf.reset_default_graph()\n",
"sess = tf.Session()\n",
"\n",
"# Data.\n",
"types = {\"inputs\": tf.int64, \"targets\": tf.int64}\n",
"shapes = {\"inputs\": tf.TensorShape([None]), \"targets\": tf.TensorShape([None])}\n",
"\n",
"train_data = tf.data.Dataset.from_generator(\n",
" generator1, output_types=types, output_shapes=shapes)\n",
"eval_data = tf.data.Dataset.from_generator(\n",
" generator2, output_types=types, output_shapes=shapes)\n",
"BATCH_SIZE = 100\n",
"train_batches = train_data.repeat().batch(BATCH_SIZE)\n",
"train_batch = train_batches.make_one_shot_iterator().get_next()\n",
"eval_batches = eval_data.repeat().batch(BATCH_SIZE)\n",
"eval_batch = eval_batches.make_one_shot_iterator().get_next()\n",
"\n",
"# Model\n",
"def model(batch, mode):\n",
" with tf.variable_scope(\"mymodel\", reuse=mode == Modes.EVAL):\n",
" # Inputs.\n",
" x, y = batch[\"inputs\"], batch[\"targets\"]\n",
" x = tf.reshape(x, [BATCH_SIZE, -1, 1])\n",
" y = tf.reshape(y, [BATCH_SIZE, -1, 1])\n",
" x_hot = tf.one_hot(x, 2) # From ints to 1-hot vectors.\n",
" y_hot = tf.one_hot(y, 2)\n",
" x = tf.layers.dense(x_hot, 32, name=\"embedding_x\")\n",
" y_emb = tf.layers.dense(y_hot, 32, name=\"embedding_y\")\n",
" positions = tf.scan(lambda a, z: tf.layers.dense(a, 32),\n",
" tf.transpose(x, [1, 0, 2, 3]))\n",
" positions = tf.transpose(positions, [1, 0, 2, 3])\n",
" x += positions\n",
" # Body.\n",
" hidden_size = 32\n",
" h = tf.layers.conv2d(x, hidden_size, (3, 1),\n",
" padding=\"same\", activation=tf.nn.relu, name=\"hidden\")\n",
" # Autoregressive part.\n",
" y_shifted = common_layers.shift_right(y_emb)\n",
" h += y_shifted\n",
" # Attention.\n",
" h = tf.expand_dims(tf.squeeze(h, axis=2), axis=1)\n",
" q = tf.layers.dense(h, 32, name=\"q\")\n",
" k = tf.layers.dense(h, 32, name=\"k\")\n",
" v = tf.layers.dense(h, 32, name=\"v\")\n",
" bias = common_attention.attention_bias_lower_triangle(\n",
" common_layers.shape_list(h)[1])\n",
" h += common_attention.dot_product_attention(q, k, v, bias)\n",
" h = tf.reshape(h, tf.shape(x))\n",
" # Logits.\n",
" o = tf.layers.conv2d(h, 2, (1, 1),\n",
" padding=\"same\", name=\"output\")\n",
" # Loss and accuracy.\n",
" loss = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=o, labels=y)\n",
" accuracy = tf.to_float(tf.equal(tf.argmax(o, axis=-1), y))\n",
" return tf.reduce_mean(loss), tf.reduce_mean(accuracy)\n",
"\n",
"# Model for train.\n",
"train_loss, train_accuracy = model(train_batch, Modes.TRAIN)\n",
"# Gradients.\n",
"train_op = tf.train.AdamOptimizer().minimize(train_loss)\n",
"# Model for eval.\n",
"eval_loss, eval_accuracy = model(eval_batch, Modes.EVAL)"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "CNhxmv4IZ4VU",
"colab_type": "code",
"outputId": "023f1b55-026c-4ac5-d5cd-b4ae3d82881d",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 781
}
},
"cell_type": "code",
"source": [
"# Train.\n",
"sess.run(tf.global_variables_initializer())\n",
"num_steps = 200\n",
"for step in range(num_steps + 1):\n",
" _, loss, accuracy = sess.run([train_op, train_loss, train_accuracy])\n",
" if step % 10 == 0:\n",
" print(\"Step %d train loss %.4f accuracy %.2f\" % (step, loss, accuracy))\n",
" loss, accuracy = sess.run([eval_loss, eval_accuracy])\n",
" print(\"Step %d eval loss %.4f accuracy %.2f\" % (step, loss, accuracy))"
],
"execution_count": 17,
"outputs": [
{
"output_type": "stream",
"text": [
"Step 0 train loss 0.9953 accuracy 0.47\n",
"Step 0 eval loss 0.8970 accuracy 0.46\n",
"Step 10 train loss 0.6047 accuracy 0.70\n",
"Step 10 eval loss 0.6131 accuracy 0.69\n",
"Step 20 train loss 0.4382 accuracy 0.84\n",
"Step 20 eval loss 0.4809 accuracy 0.76\n",
"Step 30 train loss 0.3191 accuracy 0.87\n",
"Step 30 eval loss 0.3464 accuracy 0.83\n",
"Step 40 train loss 0.1989 accuracy 0.93\n",
"Step 40 eval loss 0.6983 accuracy 0.77\n",
"Step 50 train loss 0.1759 accuracy 0.93\n",
"Step 50 eval loss 0.7074 accuracy 0.78\n",
"Step 60 train loss 0.1433 accuracy 0.94\n",
"Step 60 eval loss 0.9905 accuracy 0.79\n",
"Step 70 train loss 0.1151 accuracy 0.96\n",
"Step 70 eval loss 2.6595 accuracy 0.76\n",
"Step 80 train loss 0.0855 accuracy 0.97\n",
"Step 80 eval loss 2.6673 accuracy 0.77\n",
"Step 90 train loss 0.0642 accuracy 0.98\n",
"Step 90 eval loss 3.7616 accuracy 0.77\n",
"Step 100 train loss 0.0438 accuracy 0.98\n",
"Step 100 eval loss 4.5389 accuracy 0.76\n",
"Step 110 train loss 0.0465 accuracy 0.98\n",
"Step 110 eval loss 5.3649 accuracy 0.75\n",
"Step 120 train loss 0.0423 accuracy 0.98\n",
"Step 120 eval loss 6.9417 accuracy 0.75\n",
"Step 130 train loss 0.0387 accuracy 0.98\n",
"Step 130 eval loss 10.6566 accuracy 0.72\n",
"Step 140 train loss 0.0343 accuracy 0.98\n",
"Step 140 eval loss 11.4308 accuracy 0.70\n",
"Step 150 train loss 0.0342 accuracy 0.98\n",
"Step 150 eval loss 9.9144 accuracy 0.74\n",
"Step 160 train loss 0.0299 accuracy 0.98\n",
"Step 160 eval loss 12.8737 accuracy 0.70\n",
"Step 170 train loss 0.0290 accuracy 0.98\n",
"Step 170 eval loss 10.5681 accuracy 0.75\n",
"Step 180 train loss 0.0395 accuracy 0.98\n",
"Step 180 eval loss 7.1814 accuracy 0.76\n",
"Step 190 train loss 0.0349 accuracy 0.98\n",
"Step 190 eval loss 8.2619 accuracy 0.74\n",
"Step 200 train loss 0.0290 accuracy 0.98\n",
"Step 200 eval loss 8.7052 accuracy 0.74\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "TNC9fVcnibcH",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# Run pre-trained translation Transformer model."
]
},
{
"metadata": {
"id": "EB4MP7_y_SuQ",
"colab_type": "code",
"outputId": "e4e24363-776a-49a6-bf1f-41f898154850",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 72
}
},
"cell_type": "code",
"source": [
"# Reset.\n",
"sess.close()\n",
"tf.reset_default_graph()\n",
"sess = tf.Session()\n",
"\n",
"# Fetch the problem\n",
"# problems.available()\n",
"ende_problem = problems.problem(\"translate_ende_wmt32k\")\n",
"\n",
"# Copy the vocab file locally so we can encode inputs and decode model outputs\n",
"# All vocabs are stored on GCS\n",
"vocab_name = \"vocab.translate_ende_wmt32k.32768.subwords\"\n",
"vocab_file = os.path.join(gs_data_dir, vocab_name)\n",
"!gsutil cp {vocab_file} {data_dir}\n",
"\n",
"# Get the encoders from the problem\n",
"encoders = ende_problem.feature_encoders(data_dir)\n",
"\n",
"# Setup helper functions for encoding and decoding\n",
"def encode(input_str, output_str=None):\n",
" \"\"\"Input str to features dict, ready for inference\"\"\"\n",
" inputs = encoders[\"inputs\"].encode(input_str) + [1] # add EOS id\n",
" batch_inputs = tf.reshape(tf.constant(inputs), [1, -1, 1]) # Make it 3D.\n",
" return {\"inputs\": batch_inputs}\n",
"\n",
"def decode(integers):\n",
" \"\"\"List of ints to str\"\"\"\n",
" integers = list(np.squeeze(integers))\n",
" if 1 in integers:\n",
" integers = integers[:integers.index(1)]\n",
" return encoders[\"inputs\"].decode(np.squeeze(integers))"
],
"execution_count": 26,
"outputs": [
{
"output_type": "stream",
"text": [
"Copying gs://tensor2tensor-data/vocab.translate_ende_wmt32k.32768.subwords...\n",
"- [1 files][313.8 KiB/313.8 KiB] \n",
"Operation completed over 1 objects/313.8 KiB. \n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "g2aQW7Z6TOEu",
"colab_type": "code",
"colab": {}
},
"cell_type": "code",
"source": [
"# # Generate and view the data\n",
"# # This cell is commented out because WMT data generation can take hours\n",
"\n",
"# ende_problem.generate_data(data_dir, tmp_dir)\n",
"# example = tfe.Iterator(ende_problem.dataset(Modes.TRAIN, data_dir)).next()\n",
"# inputs = [int(x) for x in example[\"inputs\"].numpy()] # Cast to ints.\n",
"# targets = [int(x) for x in example[\"targets\"].numpy()] # Cast to ints.\n",
"\n",
"\n",
"\n",
"# # Example inputs as int-tensor.\n",
"# print(\"Inputs, encoded:\")\n",
"# print(inputs)\n",
"# print(\"Inputs, decoded:\")\n",
"# # Example inputs as a sentence.\n",
"# print(decode(inputs))\n",
"# # Example targets as int-tensor.\n",
"# print(\"Targets, encoded:\")\n",
"# print(targets)\n",
"# # Example targets as a sentence.\n",
"# print(\"Targets, decoded:\")\n",
"# print(decode(targets))"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "9l6hDQbrRUYV",
"colab_type": "code",
"outputId": "62eb9984-f2e9-4914-9af2-938302538657",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 145
}
},
"cell_type": "code",
"source": [
"# Create hparams and the model\n",
"# registry.list_models()\n",
"model_name = \"transformer\"\n",
"hparams_set = \"transformer_base\"\n",
"\n",
"hparams = trainer_lib.create_hparams(hparams_set, data_dir=data_dir, problem_name=\"translate_ende_wmt32k\")\n",
"\n",
"# NOTE: Only create the model once when restoring from a checkpoint; it's a\n",
"# Layer and so subsequent instantiations will have different variable scopes\n",
"# that will not match the checkpoint.\n",
"translate_model = registry.model(model_name)(hparams, Modes.EVAL)"
],
"execution_count": 28,
"outputs": [
{
"output_type": "stream",
"text": [
"INFO:tensorflow:Setting T2TModel mode to 'eval'\n",
"INFO:tensorflow:Setting hparams.layer_prepostprocess_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.symbol_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.label_smoothing to 0.0\n",
"INFO:tensorflow:Setting hparams.attention_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.relu_dropout to 0.0\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "FEwNUVlMYOJi",
"colab_type": "code",
"outputId": "77d95383-dde7-4738-9383-6072f037b34b",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"cell_type": "code",
"source": [
"# Copy the pretrained checkpoint locally\n",
"ckpt_name = \"transformer_ende_test\"\n",
"gs_ckpt = os.path.join(gs_ckpt_dir, ckpt_name)\n",
"!gsutil -q cp -R {gs_ckpt} {checkpoint_dir}\n",
"ckpt_path = tf.train.latest_checkpoint(os.path.join(checkpoint_dir, ckpt_name))\n",
"ckpt_path"
],
"execution_count": 29,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"u'/root/t2t/checkpoints/transformer_ende_test/model.ckpt-1421000'"
]
},
"metadata": {
"tags": []
},
"execution_count": 29
}
]
},
{
"metadata": {
"id": "3O-8E9d6TtuJ",
"colab_type": "code",
"outputId": "8e0815cb-b5c1-4ab2-f3f2-43455df813ab",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 946
}
},
"cell_type": "code",
"source": [
"# Restore and translate!\n",
"def translate(inputs):\n",
" encoded_inputs = encode(inputs)\n",
" model_output = translate_model.infer(encoded_inputs)[\"outputs\"]\n",
" tf.train.Saver().restore(sess, ckpt_path)\n",
" return decode(sess.run(model_output))\n",
"\n",
"inputs = \"The animal didn't cross the street because it was too tired\"\n",
"outputs = translate(inputs)\n",
"\n",
"print(\"Inputs: %s\" % inputs)\n",
"print(\"Outputs: %s\" % outputs)"
],
"execution_count": 30,
"outputs": [
{
"output_type": "stream",
"text": [
"INFO:tensorflow:Greedy Decoding\n",
":::MLPv0.5.0 transformer 1541816406.963202953 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/expert_utils.py:231) model_hp_layer_postprocess_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541816407.008698940 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_hidden_layers: 6\n",
":::MLPv0.5.0 transformer 1541816407.052841902 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_attention_num_heads: 8\n",
":::MLPv0.5.0 transformer 1541816407.105223894 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_attention_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541816407.320673943 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 2048, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541816407.366693020 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 512}\n",
":::MLPv0.5.0 transformer 1541816407.741213083 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541816408.042022943 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 2048, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541816408.091841936 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 512}\n",
":::MLPv0.5.0 transformer 1541816408.140681028 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541816408.439977884 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 2048, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541816408.489466906 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 512}\n",
":::MLPv0.5.0 transformer 1541816408.534550905 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541816408.824630976 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 2048, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541816408.869894028 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 512}\n",
":::MLPv0.5.0 transformer 1541816409.195842028 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541816409.489510059 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 2048, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541816409.536403894 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 512}\n",
":::MLPv0.5.0 transformer 1541816409.581830025 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541816409.875792027 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 2048, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541816409.923286915 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 512}\n",
":::MLPv0.5.0 transformer 1541816409.968816042 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541816410.104417086 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_norm: {\"hidden_size\": 512}\n",
":::MLPv0.5.0 transformer 1541816411.035595894 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/expert_utils.py:231) model_hp_layer_postprocess_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541816411.081850052 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_hidden_layers: 6\n",
":::MLPv0.5.0 transformer 1541816411.126580954 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_attention_num_heads: 8\n",
":::MLPv0.5.0 transformer 1541816411.171596050 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_attention_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541816411.610357046 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 2048, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541816411.655498981 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 512}\n",
":::MLPv0.5.0 transformer 1541816411.701941967 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541816412.204050064 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 2048, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541816412.253289938 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 512}\n",
":::MLPv0.5.0 transformer 1541816412.298547029 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541816413.108828068 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 2048, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541816413.154781103 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 512}\n",
":::MLPv0.5.0 transformer 1541816413.200869083 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541816413.709883928 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 2048, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541816413.755407095 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 512}\n",
":::MLPv0.5.0 transformer 1541816413.801271915 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541816414.311265945 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 2048, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541816414.357549906 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 512}\n",
":::MLPv0.5.0 transformer 1541816414.404336929 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541816415.239506960 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 2048, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541816415.290091038 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 512}\n",
":::MLPv0.5.0 transformer 1541816415.339240074 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541816415.472474098 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_norm: {\"hidden_size\": 512}\n",
"INFO:tensorflow:Restoring parameters from /root/t2t/checkpoints/transformer_ende_test/model.ckpt-1421000\n",
"Inputs: The animal didn't cross the street because it was too tired\n",
"Outputs: Das Tier überquerte die Straße nicht, weil es zu müde war, weil es zu müde war.\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "WIKgSZaaYrpT",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# Train Transformer on Translation "
]
},
{
"metadata": {
"id": "GJVscCdTZrAv",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 17692
},
"outputId": "ca4f2fc6-abe3-4c74-99e5-4ed35a5af587"
},
"cell_type": "code",
"source": [
"!t2t-trainer --generate_data \\\n",
" --data_dir=~/t2t_data \\\n",
" --output_dir=~/t2t_train/translate \\\n",
" --problem=translate_enfr_wmt_small32k \\\n",
" --model=transformer \\\n",
" --hparams_set=transformer_small \\\n",
" --train_steps=15000 \\\n",
" --eval_steps=10"
],
"execution_count": 31,
"outputs": [
{
"output_type": "stream",
"text": [
":::MLPv0.5.0 transformer 1541816764.552817106 (/usr/local/bin/t2t-trainer:28) run_start\n",
":::MLPv0.5.0 transformer 1541816764.553385973 (/usr/local/bin/t2t-trainer:28) run_set_random_seed\n",
"INFO:tensorflow:Generating data for translate_enfr_wmt_small32k\n",
"INFO:tensorflow:Downloading https://s3.amazonaws.com/opennmt-trainingdata/baseline-1M-enfr.tgz to /tmp/t2t_datagen/baseline-1M-enfr.tgz\n",
"100% completed\n",
"INFO:tensorflow:Successfully downloaded baseline-1M-enfr.tgz, 86644965 bytes.\n",
"INFO:tensorflow:Generating vocab file: /root/t2t_data/vocab.translate_enfr_wmt_small32k.32768.subwords\n",
"INFO:tensorflow:Generating vocab from: [['https://s3.amazonaws.com/opennmt-trainingdata/baseline-1M-enfr.tgz', ('baseline-1M-enfr/baseline-1M_train.en', 'baseline-1M-enfr/baseline-1M_train.fr')]]\n",
"INFO:tensorflow:Not downloading, file already found: /tmp/t2t_datagen/baseline-1M-enfr.tgz\n",
"INFO:tensorflow:Reading file: baseline-1M-enfr/baseline-1M_train.en\n",
"INFO:tensorflow:Reading file: baseline-1M-enfr/baseline-1M_train.fr\n",
"INFO:tensorflow:Trying min_count 500\n",
"INFO:tensorflow:Iteration 0\n",
"INFO:tensorflow:vocab_size = 1161\n",
"INFO:tensorflow:Iteration 1\n",
"INFO:tensorflow:vocab_size = 645\n",
"INFO:tensorflow:Iteration 2\n",
"INFO:tensorflow:vocab_size = 704\n",
"INFO:tensorflow:Iteration 3\n",
"INFO:tensorflow:vocab_size = 682\n",
"INFO:tensorflow:Trying min_count 250\n",
"INFO:tensorflow:Iteration 0\n",
"INFO:tensorflow:vocab_size = 2063\n",
"INFO:tensorflow:Iteration 1\n",
"INFO:tensorflow:vocab_size = 1037\n",
"INFO:tensorflow:Iteration 2\n",
"INFO:tensorflow:vocab_size = 1106\n",
"INFO:tensorflow:Iteration 3\n",
"INFO:tensorflow:vocab_size = 1093\n",
"INFO:tensorflow:Trying min_count 125\n",
"INFO:tensorflow:Iteration 0\n",
"INFO:tensorflow:vocab_size = 3732\n",
"INFO:tensorflow:Iteration 1\n",
"INFO:tensorflow:vocab_size = 1674\n",
"INFO:tensorflow:Iteration 2\n",
"INFO:tensorflow:vocab_size = 1785\n",
"INFO:tensorflow:Iteration 3\n",
"INFO:tensorflow:vocab_size = 1756\n",
"INFO:tensorflow:Trying min_count 62\n",
"INFO:tensorflow:Iteration 0\n",
"INFO:tensorflow:vocab_size = 6673\n",
"INFO:tensorflow:Iteration 1\n",
"INFO:tensorflow:vocab_size = 2713\n",
"INFO:tensorflow:Iteration 2\n",
"INFO:tensorflow:vocab_size = 2841\n",
"INFO:tensorflow:Iteration 3\n",
"INFO:tensorflow:vocab_size = 2828\n",
"INFO:tensorflow:Trying min_count 31\n",
"INFO:tensorflow:Iteration 0\n",
"INFO:tensorflow:vocab_size = 11203\n",
"INFO:tensorflow:Iteration 1\n",
"INFO:tensorflow:vocab_size = 4250\n",
"INFO:tensorflow:Iteration 2\n",
"INFO:tensorflow:vocab_size = 4402\n",
"INFO:tensorflow:Iteration 3\n",
"INFO:tensorflow:vocab_size = 4384\n",
"INFO:tensorflow:Trying min_count 15\n",
"INFO:tensorflow:Iteration 0\n",
"INFO:tensorflow:vocab_size = 18446\n",
"INFO:tensorflow:Iteration 1\n",
"INFO:tensorflow:vocab_size = 6780\n",
"INFO:tensorflow:Iteration 2\n",
"INFO:tensorflow:vocab_size = 6967\n",
"INFO:tensorflow:Iteration 3\n",
"INFO:tensorflow:vocab_size = 6946\n",
"INFO:tensorflow:Trying min_count 7\n",
"INFO:tensorflow:Iteration 0\n",
"INFO:tensorflow:vocab_size = 29911\n",
"INFO:tensorflow:Iteration 1\n",
"INFO:tensorflow:vocab_size = 10720\n",
"INFO:tensorflow:Iteration 2\n",
"INFO:tensorflow:vocab_size = 10946\n",
"INFO:tensorflow:Iteration 3\n",
"INFO:tensorflow:vocab_size = 10945\n",
"INFO:tensorflow:Trying min_count 3\n",
"INFO:tensorflow:Iteration 0\n",
"INFO:tensorflow:vocab_size = 49322\n",
"INFO:tensorflow:Iteration 1\n",
"INFO:tensorflow:vocab_size = 17487\n",
"INFO:tensorflow:Iteration 2\n",
"INFO:tensorflow:vocab_size = 17820\n",
"INFO:tensorflow:Iteration 3\n",
"INFO:tensorflow:vocab_size = 17754\n",
"INFO:tensorflow:Trying min_count 1\n",
"INFO:tensorflow:Iteration 0\n",
"INFO:tensorflow:vocab_size = 85065\n",
"INFO:tensorflow:Iteration 1\n",
"INFO:tensorflow:vocab_size = 27664\n",
"INFO:tensorflow:Iteration 2\n",
"INFO:tensorflow:vocab_size = 27664\n",
"INFO:tensorflow:Iteration 3\n",
"INFO:tensorflow:vocab_size = 27664\n",
"INFO:tensorflow:Generating case 0.\n",
"INFO:tensorflow:Generating case 100000.\n",
"INFO:tensorflow:Generating case 200000.\n",
"INFO:tensorflow:Generating case 300000.\n",
"INFO:tensorflow:Generating case 400000.\n",
"INFO:tensorflow:Generating case 500000.\n",
"INFO:tensorflow:Generating case 600000.\n",
"INFO:tensorflow:Generating case 700000.\n",
"INFO:tensorflow:Generating case 800000.\n",
"INFO:tensorflow:Generating case 900000.\n",
"INFO:tensorflow:Generating case 1000000.\n",
"INFO:tensorflow:Generated 1009163 Examples\n",
"INFO:tensorflow:Not downloading, file already found: /tmp/t2t_datagen/baseline-1M-enfr.tgz\n",
"INFO:tensorflow:Found vocab file: /root/t2t_data/vocab.translate_enfr_wmt_small32k.32768.subwords\n",
"INFO:tensorflow:Generating case 0.\n",
"INFO:tensorflow:Generated 1000 Examples\n",
"INFO:tensorflow:Shuffling data...\n",
"INFO:tensorflow:Data shuffled.\n",
"WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/trainer_lib.py:230: __init__ (from tensorflow.contrib.learn.python.learn.estimators.run_config) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"When switching to tf.estimator.Estimator, use tf.estimator.RunConfig instead.\n",
"INFO:tensorflow:Configuring DataParallelism to replicate the model.\n",
"INFO:tensorflow:schedule=continuous_train_and_eval\n",
"INFO:tensorflow:worker_gpu=1\n",
"INFO:tensorflow:sync=False\n",
"WARNING:tensorflow:Schedule=continuous_train_and_eval. Assuming that training is running on a single machine.\n",
"INFO:tensorflow:datashard_devices: ['gpu:0']\n",
"INFO:tensorflow:caching_devices: None\n",
"INFO:tensorflow:ps_devices: ['gpu:0']\n",
"INFO:tensorflow:Using config: {'_save_checkpoints_secs': None, '_num_ps_replicas': 0, '_keep_checkpoint_max': 20, '_task_type': None, '_train_distribute': None, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9a27fb3bd0>, '_tf_config': gpu_options {\n",
" per_process_gpu_memory_fraction: 1.0\n",
"}\n",
", '_protocol': None, '_save_checkpoints_steps': 1000, '_keep_checkpoint_every_n_hours': 10000, '_session_config': gpu_options {\n",
" per_process_gpu_memory_fraction: 0.95\n",
"}\n",
"allow_soft_placement: true\n",
"graph_options {\n",
" optimizer_options {\n",
" }\n",
"}\n",
", '_model_dir': '/root/t2t_train/translate', 'use_tpu': False, '_tf_random_seed': None, '_master': '', '_device_fn': None, '_num_worker_replicas': 0, '_task_id': 0, '_log_step_count_steps': 100, '_evaluation_master': '', '_eval_distribute': None, 'data_parallelism': <tensor2tensor.utils.expert_utils.Parallelism object at 0x7f9a27fb3c10>, '_environment': 'local', '_save_summary_steps': 100, 't2t_device_info': {'num_async_replicas': 1}}\n",
"WARNING:tensorflow:Estimator's model_fn (<function wrapping_model_fn at 0x7f9a0e2e0320>) includes params argument, but params are not passed to Estimator.\n",
"WARNING:tensorflow:ValidationMonitor only works with --schedule=train_and_evaluate\n",
"INFO:tensorflow:Not using Distribute Coordinator.\n",
"INFO:tensorflow:Running training and evaluation locally (non-distributed).\n",
"INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 1000 or save_checkpoints_secs None.\n",
"INFO:tensorflow:Reading data files from /root/t2t_data/translate_enfr_wmt_small32k-train*\n",
"INFO:tensorflow:partition: 0 num_data_files: 100\n",
":::MLPv0.5.0 transformer 1541817045.136044979 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/data_generators/problem.py:869) input_order\n",
"WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensor2tensor/data_generators/problem.py:653: parallel_interleave (from tensorflow.contrib.data.python.ops.interleave_ops) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Use `tf.data.experimental.parallel_interleave(...)`.\n",
"WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/sparse_ops.py:1165: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.\n",
"WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensor2tensor/data_generators/problem.py:944: bucket_by_sequence_length (from tensorflow.contrib.data.python.ops.grouping) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Use `tf.data.experimental.bucket_by_sequence_length(...)`.\n",
"INFO:tensorflow:Calling model_fn.\n",
"INFO:tensorflow:Setting T2TModel mode to 'train'\n",
":::MLPv0.5.0 transformer 1541817046.449996948 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py:213) model_hp_initializer_gain: 1.0\n",
"INFO:tensorflow:Using variable initializer: uniform_unit_scaling\n",
"INFO:tensorflow:Transforming feature 'inputs' with symbol_modality_27664_256.bottom\n",
"WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/function.py:987: calling create_op (from tensorflow.python.framework.ops) with compute_shapes is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Shapes are always computed; don't use the compute_shapes as it has no effect.\n",
"INFO:tensorflow:Transforming feature 'targets' with symbol_modality_27664_256.targets_bottom\n",
"INFO:tensorflow:Building model body\n",
":::MLPv0.5.0 transformer 1541817046.824206114 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:186) model_hp_layer_postprocess_dropout: 0.1\n",
":::MLPv0.5.0 transformer 1541817046.834860086 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_hidden_layers: 2\n",
":::MLPv0.5.0 transformer 1541817046.836236000 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_attention_num_heads: 4\n",
":::MLPv0.5.0 transformer 1541817046.837599039 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_attention_dropout: 0.1\n",
":::MLPv0.5.0 transformer 1541817047.139956951 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541817047.141402960 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541817047.142832994 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.1\n",
":::MLPv0.5.0 transformer 1541817047.556977987 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541817047.558427095 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541817047.559804916 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.1\n",
":::MLPv0.5.0 transformer 1541817047.820261002 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_norm: {\"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541817047.936907053 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:202) model_hp_layer_postprocess_dropout: 0.1\n",
":::MLPv0.5.0 transformer 1541817047.946826935 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_hidden_layers: 2\n",
":::MLPv0.5.0 transformer 1541817047.948201895 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_attention_num_heads: 4\n",
":::MLPv0.5.0 transformer 1541817047.949582100 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_attention_dropout: 0.1\n",
":::MLPv0.5.0 transformer 1541817048.503779888 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541817048.505295992 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541817048.506691933 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.1\n",
":::MLPv0.5.0 transformer 1541817049.302042961 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541817049.303599119 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541817049.305062056 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.1\n",
":::MLPv0.5.0 transformer 1541817049.403249979 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_norm: {\"hidden_size\": 256}\n",
"INFO:tensorflow:Transforming body output with symbol_modality_27664_256.top\n",
"INFO:tensorflow:Base learning rate: 2.000000\n",
"INFO:tensorflow:Trainable Variables Total size: 10771456\n",
"INFO:tensorflow:Using optimizer Adam\n",
":::MLPv0.5.0 transformer 1541817049.598567963 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/optimize.py:45) opt_name: \"Adam\"\n",
":::MLPv0.5.0 transformer 1541817049.599714041 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/optimize.py:45) opt_hp_Adam_beta1: 0.9\n",
":::MLPv0.5.0 transformer 1541817049.601015091 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/optimize.py:45) opt_hp_Adam_beta2: 0.997\n",
":::MLPv0.5.0 transformer 1541817049.602169991 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/optimize.py:45) opt_hp_Adam_epsilon: 1e-09\n",
"/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gradients_impl.py:112: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
" \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n",
"INFO:tensorflow:Done calling model_fn.\n",
"INFO:tensorflow:Create CheckpointSaverHook.\n",
"INFO:tensorflow:Graph was finalized.\n",
"2018-11-10 02:30:57.106040: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2018-11-10 02:30:57.106543: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: \n",
"name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235\n",
"pciBusID: 0000:00:04.0\n",
"totalMemory: 11.17GiB freeMemory: 10.40GiB\n",
"2018-11-10 02:30:57.106612: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0\n",
"2018-11-10 02:30:57.503615: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2018-11-10 02:30:57.503698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 \n",
"2018-11-10 02:30:57.503723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N \n",
"2018-11-10 02:30:57.504029: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
"2018-11-10 02:30:57.504090: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10869 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
"INFO:tensorflow:Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"INFO:tensorflow:Saving checkpoints for 0 into /root/t2t_train/translate/model.ckpt.\n",
"2018-11-10 02:31:18.541238: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 354 of 512\n",
"2018-11-10 02:31:22.248464: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:136] Shuffle buffer filled.\n",
"INFO:tensorflow:loss = 9.467057, step = 0\n",
"INFO:tensorflow:global_step/sec: 3.70403\n",
"INFO:tensorflow:loss = 8.327867, step = 100 (26.998 sec)\n",
"INFO:tensorflow:global_step/sec: 4.08837\n",
"INFO:tensorflow:loss = 7.011908, step = 200 (24.460 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04553\n",
"INFO:tensorflow:loss = 6.1562953, step = 300 (24.719 sec)\n",
"INFO:tensorflow:global_step/sec: 4.06825\n",
"INFO:tensorflow:loss = 5.3967633, step = 400 (24.581 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04866\n",
"INFO:tensorflow:loss = 5.2822685, step = 500 (24.700 sec)\n",
"INFO:tensorflow:global_step/sec: 4.07643\n",
"INFO:tensorflow:loss = 4.973584, step = 600 (24.532 sec)\n",
"INFO:tensorflow:global_step/sec: 4.03578\n",
"INFO:tensorflow:loss = 4.7282143, step = 700 (24.778 sec)\n",
"INFO:tensorflow:global_step/sec: 4.06317\n",
"INFO:tensorflow:loss = 4.5810947, step = 800 (24.611 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05137\n",
"INFO:tensorflow:loss = 4.4209566, step = 900 (24.684 sec)\n",
"INFO:tensorflow:Saving checkpoints for 1000 into /root/t2t_train/translate/model.ckpt.\n",
"INFO:tensorflow:Reading data files from /root/t2t_data/translate_enfr_wmt_small32k-dev*\n",
"INFO:tensorflow:partition: 0 num_data_files: 1\n",
"INFO:tensorflow:Calling model_fn.\n",
"INFO:tensorflow:Setting T2TModel mode to 'eval'\n",
"INFO:tensorflow:Setting hparams.layer_prepostprocess_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.symbol_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.label_smoothing to 0.0\n",
"INFO:tensorflow:Setting hparams.attention_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.relu_dropout to 0.0\n",
":::MLPv0.5.0 transformer 1541817333.107448101 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py:213) model_hp_initializer_gain: 1.0\n",
"INFO:tensorflow:Using variable initializer: uniform_unit_scaling\n",
"INFO:tensorflow:Transforming feature 'inputs' with symbol_modality_27664_256.bottom\n",
"INFO:tensorflow:Transforming feature 'targets' with symbol_modality_27664_256.targets_bottom\n",
"INFO:tensorflow:Building model body\n",
":::MLPv0.5.0 transformer 1541817333.491487980 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:186) model_hp_layer_postprocess_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541817333.493680954 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_hidden_layers: 2\n",
":::MLPv0.5.0 transformer 1541817333.495729923 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_attention_num_heads: 4\n",
":::MLPv0.5.0 transformer 1541817333.497785091 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_attention_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541817333.807748079 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541817333.809832096 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541817333.811878920 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541817334.220690012 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541817334.222826958 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541817334.224842072 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541817334.337461948 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_norm: {\"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541817334.468022108 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:202) model_hp_layer_postprocess_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541817334.470685005 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_hidden_layers: 2\n",
":::MLPv0.5.0 transformer 1541817334.472764015 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_attention_num_heads: 4\n",
":::MLPv0.5.0 transformer 1541817334.474889040 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_attention_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541817335.325033903 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541817335.327063084 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541817335.329124928 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541817335.957114935 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541817335.959278107 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541817335.961271048 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541817336.048753977 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_norm: {\"hidden_size\": 256}\n",
"INFO:tensorflow:Transforming body output with symbol_modality_27664_256.top\n",
"INFO:tensorflow:Done calling model_fn.\n",
"INFO:tensorflow:Starting evaluation at 2018-11-10-02:35:37\n",
"INFO:tensorflow:Graph was finalized.\n",
"2018-11-10 02:35:37.445037: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0\n",
"2018-11-10 02:35:37.445133: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2018-11-10 02:35:37.445183: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 \n",
"2018-11-10 02:35:37.445214: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N \n",
"2018-11-10 02:35:37.445446: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10869 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
"INFO:tensorflow:Restoring parameters from /root/t2t_train/translate/model.ckpt-1000\n",
"INFO:tensorflow:Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"INFO:tensorflow:Evaluation [1/10]\n",
"INFO:tensorflow:Evaluation [2/10]\n",
"INFO:tensorflow:Evaluation [3/10]\n",
"INFO:tensorflow:Evaluation [4/10]\n",
"INFO:tensorflow:Evaluation [5/10]\n",
"INFO:tensorflow:Evaluation [6/10]\n",
"INFO:tensorflow:Evaluation [7/10]\n",
"INFO:tensorflow:Evaluation [8/10]\n",
"INFO:tensorflow:Evaluation [9/10]\n",
"INFO:tensorflow:Evaluation [10/10]\n",
"INFO:tensorflow:Finished evaluation at 2018-11-10-02:35:39\n",
"INFO:tensorflow:Saving dict for global step 1000: global_step = 1000, loss = 4.305866, metrics-translate_enfr_wmt_small32k/targets/accuracy = 0.33333334, metrics-translate_enfr_wmt_small32k/targets/accuracy_per_sequence = 0.0, metrics-translate_enfr_wmt_small32k/targets/accuracy_top5 = 0.50167507, metrics-translate_enfr_wmt_small32k/targets/approx_bleu_score = 0.048576087, metrics-translate_enfr_wmt_small32k/targets/neg_log_perplexity = -4.3616643, metrics-translate_enfr_wmt_small32k/targets/rouge_2_fscore = 0.14512943, metrics-translate_enfr_wmt_small32k/targets/rouge_L_fscore = 0.3745911\n",
"INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /root/t2t_train/translate/model.ckpt-1000\n",
"INFO:tensorflow:global_step/sec: 2.92135\n",
"INFO:tensorflow:loss = 3.9329858, step = 1000 (34.230 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05738\n",
"INFO:tensorflow:loss = 4.1354976, step = 1100 (24.647 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05987\n",
"INFO:tensorflow:loss = 4.000768, step = 1200 (24.631 sec)\n",
"INFO:tensorflow:global_step/sec: 4.02875\n",
"INFO:tensorflow:loss = 3.6514008, step = 1300 (24.822 sec)\n",
"INFO:tensorflow:global_step/sec: 4.02731\n",
"INFO:tensorflow:loss = 3.8132305, step = 1400 (24.831 sec)\n",
"INFO:tensorflow:global_step/sec: 4.07829\n",
"INFO:tensorflow:loss = 3.7230632, step = 1500 (24.520 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04819\n",
"INFO:tensorflow:loss = 3.5000157, step = 1600 (24.702 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05096\n",
"INFO:tensorflow:loss = 3.5503862, step = 1700 (24.686 sec)\n",
"INFO:tensorflow:global_step/sec: 4.06856\n",
"INFO:tensorflow:loss = 3.564161, step = 1800 (24.578 sec)\n",
"INFO:tensorflow:global_step/sec: 4.01588\n",
"INFO:tensorflow:loss = 3.2854428, step = 1900 (24.901 sec)\n",
"INFO:tensorflow:Saving checkpoints for 2000 into /root/t2t_train/translate/model.ckpt.\n",
"INFO:tensorflow:Skip the current checkpoint eval due to throttle secs (600 secs).\n",
"INFO:tensorflow:global_step/sec: 3.86546\n",
"INFO:tensorflow:loss = 3.295119, step = 2000 (25.870 sec)\n",
"INFO:tensorflow:global_step/sec: 4.10893\n",
"INFO:tensorflow:loss = 3.1991017, step = 2100 (24.338 sec)\n",
"INFO:tensorflow:global_step/sec: 4.01867\n",
"INFO:tensorflow:loss = 3.2506897, step = 2200 (24.884 sec)\n",
"INFO:tensorflow:global_step/sec: 4.06856\n",
"INFO:tensorflow:loss = 3.12569, step = 2300 (24.579 sec)\n",
"INFO:tensorflow:global_step/sec: 4.033\n",
"INFO:tensorflow:loss = 3.0059047, step = 2400 (24.795 sec)\n",
"INFO:tensorflow:global_step/sec: 4.03862\n",
"INFO:tensorflow:loss = 2.834777, step = 2500 (24.761 sec)\n",
"INFO:tensorflow:global_step/sec: 4.03598\n",
"INFO:tensorflow:loss = 2.8298156, step = 2600 (24.777 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05118\n",
"INFO:tensorflow:loss = 2.8108127, step = 2700 (24.684 sec)\n",
"INFO:tensorflow:global_step/sec: 4.043\n",
"INFO:tensorflow:loss = 2.8584363, step = 2800 (24.734 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04873\n",
"INFO:tensorflow:loss = 2.89839, step = 2900 (24.699 sec)\n",
"INFO:tensorflow:Saving checkpoints for 3000 into /root/t2t_train/translate/model.ckpt.\n",
"INFO:tensorflow:Skip the current checkpoint eval due to throttle secs (600 secs).\n",
"INFO:tensorflow:global_step/sec: 3.93728\n",
"INFO:tensorflow:loss = 2.973921, step = 3000 (25.398 sec)\n",
"INFO:tensorflow:global_step/sec: 4.02414\n",
"INFO:tensorflow:loss = 2.714225, step = 3100 (24.850 sec)\n",
"INFO:tensorflow:global_step/sec: 4.07647\n",
"INFO:tensorflow:loss = 2.6924918, step = 3200 (24.531 sec)\n",
"INFO:tensorflow:global_step/sec: 4.03677\n",
"INFO:tensorflow:loss = 2.6270318, step = 3300 (24.773 sec)\n",
"INFO:tensorflow:global_step/sec: 4.0509\n",
"INFO:tensorflow:loss = 2.5606387, step = 3400 (24.685 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05653\n",
"INFO:tensorflow:loss = 2.708626, step = 3500 (24.655 sec)\n",
"INFO:tensorflow:global_step/sec: 4.0533\n",
"INFO:tensorflow:loss = 2.26777, step = 3600 (24.668 sec)\n",
"INFO:tensorflow:global_step/sec: 4.07834\n",
"INFO:tensorflow:loss = 1.9720646, step = 3700 (24.520 sec)\n",
"INFO:tensorflow:global_step/sec: 4.07782\n",
"INFO:tensorflow:loss = 2.804067, step = 3800 (24.523 sec)\n",
"INFO:tensorflow:global_step/sec: 4.0575\n",
"INFO:tensorflow:loss = 2.3475785, step = 3900 (24.646 sec)\n",
"INFO:tensorflow:Saving checkpoints for 4000 into /root/t2t_train/translate/model.ckpt.\n",
"INFO:tensorflow:Reading data files from /root/t2t_data/translate_enfr_wmt_small32k-dev*\n",
"INFO:tensorflow:partition: 0 num_data_files: 1\n",
"INFO:tensorflow:Calling model_fn.\n",
"INFO:tensorflow:Setting T2TModel mode to 'eval'\n",
"INFO:tensorflow:Setting hparams.layer_prepostprocess_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.symbol_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.label_smoothing to 0.0\n",
"INFO:tensorflow:Setting hparams.attention_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.relu_dropout to 0.0\n",
":::MLPv0.5.0 transformer 1541818084.965792894 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py:213) model_hp_initializer_gain: 1.0\n",
"INFO:tensorflow:Using variable initializer: uniform_unit_scaling\n",
"INFO:tensorflow:Transforming feature 'inputs' with symbol_modality_27664_256.bottom\n",
"INFO:tensorflow:Transforming feature 'targets' with symbol_modality_27664_256.targets_bottom\n",
"INFO:tensorflow:Building model body\n",
":::MLPv0.5.0 transformer 1541818085.724955082 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:186) model_hp_layer_postprocess_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541818085.727355957 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_hidden_layers: 2\n",
":::MLPv0.5.0 transformer 1541818085.729433060 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_attention_num_heads: 4\n",
":::MLPv0.5.0 transformer 1541818085.731327057 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_attention_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541818086.067890882 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541818086.070430040 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541818086.072455883 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541818086.474518061 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541818086.476706028 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541818086.478760004 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541818086.589759111 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_norm: {\"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541818086.712661028 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:202) model_hp_layer_postprocess_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541818086.714766026 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_hidden_layers: 2\n",
":::MLPv0.5.0 transformer 1541818086.716799021 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_attention_num_heads: 4\n",
":::MLPv0.5.0 transformer 1541818086.718868017 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_attention_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541818087.276854038 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541818087.279017925 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541818087.281033993 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541818088.232291937 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541818088.234427929 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541818088.236588955 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541818088.323031902 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_norm: {\"hidden_size\": 256}\n",
"INFO:tensorflow:Transforming body output with symbol_modality_27664_256.top\n",
"INFO:tensorflow:Done calling model_fn.\n",
"INFO:tensorflow:Starting evaluation at 2018-11-10-02:48:09\n",
"INFO:tensorflow:Graph was finalized.\n",
"2018-11-10 02:48:09.617917: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0\n",
"2018-11-10 02:48:09.618032: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2018-11-10 02:48:09.618063: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 \n",
"2018-11-10 02:48:09.618091: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N \n",
"2018-11-10 02:48:09.618346: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10869 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
"INFO:tensorflow:Restoring parameters from /root/t2t_train/translate/model.ckpt-4000\n",
"INFO:tensorflow:Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"INFO:tensorflow:Evaluation [1/10]\n",
"INFO:tensorflow:Evaluation [2/10]\n",
"INFO:tensorflow:Evaluation [3/10]\n",
"INFO:tensorflow:Evaluation [4/10]\n",
"INFO:tensorflow:Evaluation [5/10]\n",
"INFO:tensorflow:Evaluation [6/10]\n",
"INFO:tensorflow:Evaluation [7/10]\n",
"INFO:tensorflow:Evaluation [8/10]\n",
"INFO:tensorflow:Evaluation [9/10]\n",
"INFO:tensorflow:Evaluation [10/10]\n",
"INFO:tensorflow:Finished evaluation at 2018-11-10-02:48:11\n",
"INFO:tensorflow:Saving dict for global step 4000: global_step = 4000, loss = 1.9216549, metrics-translate_enfr_wmt_small32k/targets/accuracy = 0.60510886, metrics-translate_enfr_wmt_small32k/targets/accuracy_per_sequence = 0.027777778, metrics-translate_enfr_wmt_small32k/targets/accuracy_top5 = 0.80025125, metrics-translate_enfr_wmt_small32k/targets/approx_bleu_score = 0.30492762, metrics-translate_enfr_wmt_small32k/targets/neg_log_perplexity = -2.0240185, metrics-translate_enfr_wmt_small32k/targets/rouge_2_fscore = 0.39612994, metrics-translate_enfr_wmt_small32k/targets/rouge_L_fscore = 0.59713376\n",
"INFO:tensorflow:Saving 'checkpoint_path' summary for global step 4000: /root/t2t_train/translate/model.ckpt-4000\n",
"INFO:tensorflow:global_step/sec: 2.9884\n",
"INFO:tensorflow:loss = 2.462601, step = 4000 (33.462 sec)\n",
"INFO:tensorflow:global_step/sec: 4.06274\n",
"INFO:tensorflow:loss = 2.5394523, step = 4100 (24.614 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04627\n",
"INFO:tensorflow:loss = 2.1326315, step = 4200 (24.714 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05189\n",
"INFO:tensorflow:loss = 2.48495, step = 4300 (24.680 sec)\n",
"INFO:tensorflow:global_step/sec: 4.02561\n",
"INFO:tensorflow:loss = 2.3976479, step = 4400 (24.841 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05706\n",
"INFO:tensorflow:loss = 2.5906184, step = 4500 (24.649 sec)\n",
"INFO:tensorflow:global_step/sec: 4.02257\n",
"INFO:tensorflow:loss = 2.3204334, step = 4600 (24.859 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05126\n",
"INFO:tensorflow:loss = 2.2512567, step = 4700 (24.683 sec)\n",
"INFO:tensorflow:global_step/sec: 4.09533\n",
"INFO:tensorflow:loss = 1.6915437, step = 4800 (24.418 sec)\n",
"INFO:tensorflow:global_step/sec: 4.031\n",
"INFO:tensorflow:loss = 2.2468112, step = 4900 (24.807 sec)\n",
"INFO:tensorflow:Saving checkpoints for 5000 into /root/t2t_train/translate/model.ckpt.\n",
"INFO:tensorflow:Skip the current checkpoint eval due to throttle secs (600 secs).\n",
"INFO:tensorflow:global_step/sec: 3.86869\n",
"INFO:tensorflow:loss = 1.6911397, step = 5000 (25.849 sec)\n",
"INFO:tensorflow:global_step/sec: 4.06075\n",
"INFO:tensorflow:loss = 2.2577803, step = 5100 (24.626 sec)\n",
"INFO:tensorflow:global_step/sec: 4.07183\n",
"INFO:tensorflow:loss = 2.1528697, step = 5200 (24.559 sec)\n",
"INFO:tensorflow:global_step/sec: 4.0267\n",
"INFO:tensorflow:loss = 2.417025, step = 5300 (24.834 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04155\n",
"INFO:tensorflow:loss = 2.0929453, step = 5400 (24.743 sec)\n",
"INFO:tensorflow:global_step/sec: 4.06641\n",
"INFO:tensorflow:loss = 2.1303144, step = 5500 (24.592 sec)\n",
"INFO:tensorflow:global_step/sec: 4.08202\n",
"INFO:tensorflow:loss = 2.2132716, step = 5600 (24.498 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04924\n",
"INFO:tensorflow:loss = 1.913886, step = 5700 (24.696 sec)\n",
"INFO:tensorflow:global_step/sec: 4.03903\n",
"INFO:tensorflow:loss = 2.2120955, step = 5800 (24.758 sec)\n",
"INFO:tensorflow:global_step/sec: 4.09679\n",
"INFO:tensorflow:loss = 2.274867, step = 5900 (24.410 sec)\n",
"INFO:tensorflow:Saving checkpoints for 6000 into /root/t2t_train/translate/model.ckpt.\n",
"INFO:tensorflow:Skip the current checkpoint eval due to throttle secs (600 secs).\n",
"INFO:tensorflow:global_step/sec: 3.80445\n",
"INFO:tensorflow:loss = 2.3264813, step = 6000 (26.287 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04614\n",
"INFO:tensorflow:loss = 2.131481, step = 6100 (24.713 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04827\n",
"INFO:tensorflow:loss = 1.9493632, step = 6200 (24.702 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05505\n",
"INFO:tensorflow:loss = 1.9686915, step = 6300 (24.661 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05902\n",
"INFO:tensorflow:loss = 2.06327, step = 6400 (24.636 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04184\n",
"INFO:tensorflow:loss = 2.1095023, step = 6500 (24.742 sec)\n",
"INFO:tensorflow:global_step/sec: 4.08044\n",
"INFO:tensorflow:loss = 2.2507858, step = 6600 (24.507 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04584\n",
"INFO:tensorflow:loss = 2.0669193, step = 6700 (24.718 sec)\n",
"INFO:tensorflow:global_step/sec: 4.02749\n",
"INFO:tensorflow:loss = 1.7660891, step = 6800 (24.829 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04276\n",
"INFO:tensorflow:loss = 1.9465747, step = 6900 (24.736 sec)\n",
"INFO:tensorflow:Saving checkpoints for 7000 into /root/t2t_train/translate/model.ckpt.\n",
"INFO:tensorflow:Reading data files from /root/t2t_data/translate_enfr_wmt_small32k-dev*\n",
"INFO:tensorflow:partition: 0 num_data_files: 1\n",
"INFO:tensorflow:Calling model_fn.\n",
"INFO:tensorflow:Setting T2TModel mode to 'eval'\n",
"INFO:tensorflow:Setting hparams.layer_prepostprocess_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.symbol_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.label_smoothing to 0.0\n",
"INFO:tensorflow:Setting hparams.attention_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.relu_dropout to 0.0\n",
":::MLPv0.5.0 transformer 1541818836.825932026 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py:213) model_hp_initializer_gain: 1.0\n",
"INFO:tensorflow:Using variable initializer: uniform_unit_scaling\n",
"INFO:tensorflow:Transforming feature 'inputs' with symbol_modality_27664_256.bottom\n",
"INFO:tensorflow:Transforming feature 'targets' with symbol_modality_27664_256.targets_bottom\n",
"INFO:tensorflow:Building model body\n",
":::MLPv0.5.0 transformer 1541818837.194365025 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:186) model_hp_layer_postprocess_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541818837.196631908 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_hidden_layers: 2\n",
":::MLPv0.5.0 transformer 1541818837.198563099 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_attention_num_heads: 4\n",
":::MLPv0.5.0 transformer 1541818837.200685978 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_attention_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541818837.500430107 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541818837.502458096 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541818837.504445076 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541818837.917840958 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541818837.920069933 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541818837.922117949 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541818838.030708075 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_norm: {\"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541818838.159888983 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:202) model_hp_layer_postprocess_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541818838.161999941 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_hidden_layers: 2\n",
":::MLPv0.5.0 transformer 1541818838.164275885 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_attention_num_heads: 4\n",
":::MLPv0.5.0 transformer 1541818838.166384935 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_attention_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541818839.007117987 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541818839.009202003 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541818839.011311054 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541818839.655536890 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541818839.657636881 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541818839.659749985 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541818839.748116970 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_norm: {\"hidden_size\": 256}\n",
"INFO:tensorflow:Transforming body output with symbol_modality_27664_256.top\n",
"INFO:tensorflow:Done calling model_fn.\n",
"INFO:tensorflow:Starting evaluation at 2018-11-10-03:00:40\n",
"INFO:tensorflow:Graph was finalized.\n",
"2018-11-10 03:00:41.063892: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0\n",
"2018-11-10 03:00:41.063981: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2018-11-10 03:00:41.064010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 \n",
"2018-11-10 03:00:41.064038: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N \n",
"2018-11-10 03:00:41.064274: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10869 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
"INFO:tensorflow:Restoring parameters from /root/t2t_train/translate/model.ckpt-7000\n",
"INFO:tensorflow:Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"INFO:tensorflow:Evaluation [1/10]\n",
"INFO:tensorflow:Evaluation [2/10]\n",
"INFO:tensorflow:Evaluation [3/10]\n",
"INFO:tensorflow:Evaluation [4/10]\n",
"INFO:tensorflow:Evaluation [5/10]\n",
"INFO:tensorflow:Evaluation [6/10]\n",
"INFO:tensorflow:Evaluation [7/10]\n",
"INFO:tensorflow:Evaluation [8/10]\n",
"INFO:tensorflow:Evaluation [9/10]\n",
"INFO:tensorflow:Evaluation [10/10]\n",
"INFO:tensorflow:Finished evaluation at 2018-11-10-03:00:43\n",
"INFO:tensorflow:Saving dict for global step 7000: global_step = 7000, loss = 1.5866534, metrics-translate_enfr_wmt_small32k/targets/accuracy = 0.65201, metrics-translate_enfr_wmt_small32k/targets/accuracy_per_sequence = 0.060185187, metrics-translate_enfr_wmt_small32k/targets/accuracy_top5 = 0.8400335, metrics-translate_enfr_wmt_small32k/targets/approx_bleu_score = 0.37232235, metrics-translate_enfr_wmt_small32k/targets/neg_log_perplexity = -1.6673156, metrics-translate_enfr_wmt_small32k/targets/rouge_2_fscore = 0.4513235, metrics-translate_enfr_wmt_small32k/targets/rouge_L_fscore = 0.63478863\n",
"INFO:tensorflow:Saving 'checkpoint_path' summary for global step 7000: /root/t2t_train/translate/model.ckpt-7000\n",
"INFO:tensorflow:global_step/sec: 3.02454\n",
"INFO:tensorflow:loss = 2.1321363, step = 7000 (33.062 sec)\n",
"INFO:tensorflow:global_step/sec: 4.07066\n",
"INFO:tensorflow:loss = 2.0522125, step = 7100 (24.566 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04356\n",
"INFO:tensorflow:loss = 2.0980835, step = 7200 (24.731 sec)\n",
"INFO:tensorflow:global_step/sec: 4.06237\n",
"INFO:tensorflow:loss = 2.0467265, step = 7300 (24.616 sec)\n",
"INFO:tensorflow:global_step/sec: 4.09287\n",
"INFO:tensorflow:loss = 1.9539516, step = 7400 (24.432 sec)\n",
"INFO:tensorflow:global_step/sec: 4.06651\n",
"INFO:tensorflow:loss = 2.4500518, step = 7500 (24.591 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05728\n",
"INFO:tensorflow:loss = 2.1298695, step = 7600 (24.647 sec)\n",
"INFO:tensorflow:global_step/sec: 4.0506\n",
"INFO:tensorflow:loss = 1.8243495, step = 7700 (24.688 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04017\n",
"INFO:tensorflow:loss = 1.988086, step = 7800 (24.752 sec)\n",
"INFO:tensorflow:global_step/sec: 4.06202\n",
"INFO:tensorflow:loss = 1.5627835, step = 7900 (24.618 sec)\n",
"INFO:tensorflow:Saving checkpoints for 8000 into /root/t2t_train/translate/model.ckpt.\n",
"INFO:tensorflow:Skip the current checkpoint eval due to throttle secs (600 secs).\n",
"INFO:tensorflow:global_step/sec: 3.83612\n",
"INFO:tensorflow:loss = 1.9024932, step = 8000 (26.075 sec)\n",
"INFO:tensorflow:global_step/sec: 4.08988\n",
"INFO:tensorflow:loss = 1.9240679, step = 8100 (24.445 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05334\n",
"INFO:tensorflow:loss = 2.0336208, step = 8200 (24.670 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05\n",
"INFO:tensorflow:loss = 1.7850604, step = 8300 (24.692 sec)\n",
"INFO:tensorflow:global_step/sec: 4.056\n",
"INFO:tensorflow:loss = 1.9347731, step = 8400 (24.655 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05236\n",
"INFO:tensorflow:loss = 1.6150254, step = 8500 (24.677 sec)\n",
"INFO:tensorflow:global_step/sec: 4.07252\n",
"INFO:tensorflow:loss = 2.2105553, step = 8600 (24.555 sec)\n",
"INFO:tensorflow:global_step/sec: 4.03762\n",
"INFO:tensorflow:loss = 2.0202792, step = 8700 (24.767 sec)\n",
"INFO:tensorflow:global_step/sec: 4.06947\n",
"INFO:tensorflow:loss = 1.8730527, step = 8800 (24.573 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04949\n",
"INFO:tensorflow:loss = 1.9710113, step = 8900 (24.695 sec)\n",
"INFO:tensorflow:Saving checkpoints for 9000 into /root/t2t_train/translate/model.ckpt.\n",
"INFO:tensorflow:Skip the current checkpoint eval due to throttle secs (600 secs).\n",
"INFO:tensorflow:global_step/sec: 3.85487\n",
"INFO:tensorflow:loss = 2.0631185, step = 9000 (25.941 sec)\n",
"INFO:tensorflow:global_step/sec: 4.06103\n",
"INFO:tensorflow:loss = 1.7741756, step = 9100 (24.625 sec)\n",
"INFO:tensorflow:global_step/sec: 4.02585\n",
"INFO:tensorflow:loss = 1.8898069, step = 9200 (24.839 sec)\n",
"INFO:tensorflow:global_step/sec: 4.03563\n",
"INFO:tensorflow:loss = 1.8677891, step = 9300 (24.780 sec)\n",
"INFO:tensorflow:global_step/sec: 4.02993\n",
"INFO:tensorflow:loss = 1.7967162, step = 9400 (24.814 sec)\n",
"INFO:tensorflow:global_step/sec: 4.00863\n",
"INFO:tensorflow:loss = 1.7628547, step = 9500 (24.946 sec)\n",
"INFO:tensorflow:global_step/sec: 4.0127\n",
"INFO:tensorflow:loss = 1.6898324, step = 9600 (24.921 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05966\n",
"INFO:tensorflow:loss = 1.3610109, step = 9700 (24.633 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04678\n",
"INFO:tensorflow:loss = 1.7353562, step = 9800 (24.711 sec)\n",
"INFO:tensorflow:global_step/sec: 4.02538\n",
"INFO:tensorflow:loss = 1.8497021, step = 9900 (24.843 sec)\n",
"INFO:tensorflow:Saving checkpoints for 10000 into /root/t2t_train/translate/model.ckpt.\n",
"INFO:tensorflow:Reading data files from /root/t2t_data/translate_enfr_wmt_small32k-dev*\n",
"INFO:tensorflow:partition: 0 num_data_files: 1\n",
"INFO:tensorflow:Calling model_fn.\n",
"INFO:tensorflow:Setting T2TModel mode to 'eval'\n",
"INFO:tensorflow:Setting hparams.layer_prepostprocess_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.symbol_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.label_smoothing to 0.0\n",
"INFO:tensorflow:Setting hparams.attention_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.relu_dropout to 0.0\n",
":::MLPv0.5.0 transformer 1541819588.859472036 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py:213) model_hp_initializer_gain: 1.0\n",
"INFO:tensorflow:Using variable initializer: uniform_unit_scaling\n",
"INFO:tensorflow:Transforming feature 'inputs' with symbol_modality_27664_256.bottom\n",
"INFO:tensorflow:Transforming feature 'targets' with symbol_modality_27664_256.targets_bottom\n",
"INFO:tensorflow:Building model body\n",
":::MLPv0.5.0 transformer 1541819589.256143093 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:186) model_hp_layer_postprocess_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541819589.258685112 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_hidden_layers: 2\n",
":::MLPv0.5.0 transformer 1541819589.260783911 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_attention_num_heads: 4\n",
":::MLPv0.5.0 transformer 1541819589.262870073 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_attention_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541819589.567368984 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541819589.569406033 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541819589.571448088 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541819589.968128920 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541819589.970307112 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541819589.972413063 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541819590.081412077 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_norm: {\"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541819590.218425035 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:202) model_hp_layer_postprocess_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541819590.220664978 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_hidden_layers: 2\n",
":::MLPv0.5.0 transformer 1541819590.222631931 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_attention_num_heads: 4\n",
":::MLPv0.5.0 transformer 1541819590.224656105 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_attention_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541819590.758654118 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541819590.760823011 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541819590.762934923 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541819591.703013897 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541819591.705110073 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541819591.707179070 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541819591.795419931 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_norm: {\"hidden_size\": 256}\n",
"INFO:tensorflow:Transforming body output with symbol_modality_27664_256.top\n",
"INFO:tensorflow:Done calling model_fn.\n",
"INFO:tensorflow:Starting evaluation at 2018-11-10-03:13:12\n",
"INFO:tensorflow:Graph was finalized.\n",
"2018-11-10 03:13:13.115401: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0\n",
"2018-11-10 03:13:13.115507: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2018-11-10 03:13:13.115538: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 \n",
"2018-11-10 03:13:13.115566: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N \n",
"2018-11-10 03:13:13.115813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10869 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
"INFO:tensorflow:Restoring parameters from /root/t2t_train/translate/model.ckpt-10000\n",
"INFO:tensorflow:Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"INFO:tensorflow:Evaluation [1/10]\n",
"INFO:tensorflow:Evaluation [2/10]\n",
"INFO:tensorflow:Evaluation [3/10]\n",
"INFO:tensorflow:Evaluation [4/10]\n",
"INFO:tensorflow:Evaluation [5/10]\n",
"INFO:tensorflow:Evaluation [6/10]\n",
"INFO:tensorflow:Evaluation [7/10]\n",
"INFO:tensorflow:Evaluation [8/10]\n",
"INFO:tensorflow:Evaluation [9/10]\n",
"INFO:tensorflow:Evaluation [10/10]\n",
"INFO:tensorflow:Finished evaluation at 2018-11-10-03:13:15\n",
"INFO:tensorflow:Saving dict for global step 10000: global_step = 10000, loss = 1.4206655, metrics-translate_enfr_wmt_small32k/targets/accuracy = 0.6792295, metrics-translate_enfr_wmt_small32k/targets/accuracy_per_sequence = 0.06944445, metrics-translate_enfr_wmt_small32k/targets/accuracy_top5 = 0.86767167, metrics-translate_enfr_wmt_small32k/targets/approx_bleu_score = 0.39912757, metrics-translate_enfr_wmt_small32k/targets/neg_log_perplexity = -1.4911711, metrics-translate_enfr_wmt_small32k/targets/rouge_2_fscore = 0.47228608, metrics-translate_enfr_wmt_small32k/targets/rouge_L_fscore = 0.6594284\n",
"INFO:tensorflow:Saving 'checkpoint_path' summary for global step 10000: /root/t2t_train/translate/model.ckpt-10000\n",
"INFO:tensorflow:global_step/sec: 2.97957\n",
"INFO:tensorflow:loss = 1.6939191, step = 10000 (33.561 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05294\n",
"INFO:tensorflow:loss = 1.9482076, step = 10100 (24.674 sec)\n",
"INFO:tensorflow:global_step/sec: 4.03852\n",
"INFO:tensorflow:loss = 1.7480322, step = 10200 (24.767 sec)\n",
"INFO:tensorflow:global_step/sec: 4.03879\n",
"INFO:tensorflow:loss = 1.6599189, step = 10300 (24.754 sec)\n",
"INFO:tensorflow:global_step/sec: 4.02008\n",
"INFO:tensorflow:loss = 1.6528381, step = 10400 (24.875 sec)\n",
"INFO:tensorflow:global_step/sec: 4.07678\n",
"INFO:tensorflow:loss = 1.6492587, step = 10500 (24.529 sec)\n",
"INFO:tensorflow:global_step/sec: 4.0502\n",
"INFO:tensorflow:loss = 1.7703876, step = 10600 (24.690 sec)\n",
"INFO:tensorflow:global_step/sec: 4.08699\n",
"INFO:tensorflow:loss = 1.6995385, step = 10700 (24.469 sec)\n",
"INFO:tensorflow:global_step/sec: 4.06166\n",
"INFO:tensorflow:loss = 1.7579752, step = 10800 (24.619 sec)\n",
"INFO:tensorflow:global_step/sec: 4.08871\n",
"INFO:tensorflow:loss = 1.8328538, step = 10900 (24.458 sec)\n",
"INFO:tensorflow:Saving checkpoints for 11000 into /root/t2t_train/translate/model.ckpt.\n",
"INFO:tensorflow:Skip the current checkpoint eval due to throttle secs (600 secs).\n",
"INFO:tensorflow:global_step/sec: 3.88923\n",
"INFO:tensorflow:loss = 1.4817553, step = 11000 (25.712 sec)\n",
"INFO:tensorflow:global_step/sec: 4.01814\n",
"INFO:tensorflow:loss = 2.0036612, step = 11100 (24.888 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05372\n",
"INFO:tensorflow:loss = 1.6879854, step = 11200 (24.668 sec)\n",
"INFO:tensorflow:global_step/sec: 4.06173\n",
"INFO:tensorflow:loss = 1.2620245, step = 11300 (24.620 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04984\n",
"INFO:tensorflow:loss = 1.4883475, step = 11400 (24.692 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04511\n",
"INFO:tensorflow:loss = 1.7708547, step = 11500 (24.721 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05253\n",
"INFO:tensorflow:loss = 1.6787874, step = 11600 (24.677 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05729\n",
"INFO:tensorflow:loss = 1.703857, step = 11700 (24.646 sec)\n",
"INFO:tensorflow:global_step/sec: 4.06694\n",
"INFO:tensorflow:loss = 1.8798635, step = 11800 (24.589 sec)\n",
"INFO:tensorflow:global_step/sec: 4.08702\n",
"INFO:tensorflow:loss = 1.6112572, step = 11900 (24.467 sec)\n",
"INFO:tensorflow:Saving checkpoints for 12000 into /root/t2t_train/translate/model.ckpt.\n",
"INFO:tensorflow:Skip the current checkpoint eval due to throttle secs (600 secs).\n",
"INFO:tensorflow:global_step/sec: 3.83775\n",
"INFO:tensorflow:loss = 1.8080653, step = 12000 (26.056 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04241\n",
"INFO:tensorflow:loss = 1.5517294, step = 12100 (24.739 sec)\n",
"INFO:tensorflow:global_step/sec: 3.9972\n",
"INFO:tensorflow:loss = 1.8055048, step = 12200 (25.017 sec)\n",
"INFO:tensorflow:global_step/sec: 4.07323\n",
"INFO:tensorflow:loss = 1.5828784, step = 12300 (24.551 sec)\n",
"INFO:tensorflow:global_step/sec: 4.09013\n",
"INFO:tensorflow:loss = 1.8241028, step = 12400 (24.449 sec)\n",
"INFO:tensorflow:global_step/sec: 4.02507\n",
"INFO:tensorflow:loss = 1.7099892, step = 12500 (24.844 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05965\n",
"INFO:tensorflow:loss = 1.6514162, step = 12600 (24.633 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04959\n",
"INFO:tensorflow:loss = 1.9844031, step = 12700 (24.694 sec)\n",
"INFO:tensorflow:global_step/sec: 4.03533\n",
"INFO:tensorflow:loss = 1.4861472, step = 12800 (24.781 sec)\n",
"INFO:tensorflow:global_step/sec: 4.03901\n",
"INFO:tensorflow:loss = 1.5681537, step = 12900 (24.759 sec)\n",
"INFO:tensorflow:Saving checkpoints for 13000 into /root/t2t_train/translate/model.ckpt.\n",
"INFO:tensorflow:Reading data files from /root/t2t_data/translate_enfr_wmt_small32k-dev*\n",
"INFO:tensorflow:partition: 0 num_data_files: 1\n",
"INFO:tensorflow:Calling model_fn.\n",
"INFO:tensorflow:Setting T2TModel mode to 'eval'\n",
"INFO:tensorflow:Setting hparams.layer_prepostprocess_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.symbol_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.label_smoothing to 0.0\n",
"INFO:tensorflow:Setting hparams.attention_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.relu_dropout to 0.0\n",
":::MLPv0.5.0 transformer 1541820340.068326950 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py:213) model_hp_initializer_gain: 1.0\n",
"INFO:tensorflow:Using variable initializer: uniform_unit_scaling\n",
"INFO:tensorflow:Transforming feature 'inputs' with symbol_modality_27664_256.bottom\n",
"INFO:tensorflow:Transforming feature 'targets' with symbol_modality_27664_256.targets_bottom\n",
"INFO:tensorflow:Building model body\n",
":::MLPv0.5.0 transformer 1541820340.471555948 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:186) model_hp_layer_postprocess_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820340.473917007 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_hidden_layers: 2\n",
":::MLPv0.5.0 transformer 1541820340.475946903 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_attention_num_heads: 4\n",
":::MLPv0.5.0 transformer 1541820340.477915049 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_attention_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820340.786506891 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541820340.788578987 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541820340.790587902 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820341.481549978 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541820341.483705997 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541820341.485802889 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820341.593667984 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_norm: {\"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541820341.716634989 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:202) model_hp_layer_postprocess_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820341.718744040 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_hidden_layers: 2\n",
":::MLPv0.5.0 transformer 1541820341.720756054 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_attention_num_heads: 4\n",
":::MLPv0.5.0 transformer 1541820341.722750902 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_attention_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820342.262821913 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541820342.265031099 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541820342.267077923 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820342.895304918 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541820342.897401094 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541820342.899432898 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820342.986167908 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_norm: {\"hidden_size\": 256}\n",
"INFO:tensorflow:Transforming body output with symbol_modality_27664_256.top\n",
"INFO:tensorflow:Done calling model_fn.\n",
"INFO:tensorflow:Starting evaluation at 2018-11-10-03:25:44\n",
"INFO:tensorflow:Graph was finalized.\n",
"2018-11-10 03:25:44.612789: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0\n",
"2018-11-10 03:25:44.612858: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2018-11-10 03:25:44.612928: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 \n",
"2018-11-10 03:25:44.612957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N \n",
"2018-11-10 03:25:44.613220: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10869 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
"INFO:tensorflow:Restoring parameters from /root/t2t_train/translate/model.ckpt-13000\n",
"INFO:tensorflow:Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"INFO:tensorflow:Evaluation [1/10]\n",
"INFO:tensorflow:Evaluation [2/10]\n",
"INFO:tensorflow:Evaluation [3/10]\n",
"INFO:tensorflow:Evaluation [4/10]\n",
"INFO:tensorflow:Evaluation [5/10]\n",
"INFO:tensorflow:Evaluation [6/10]\n",
"INFO:tensorflow:Evaluation [7/10]\n",
"INFO:tensorflow:Evaluation [8/10]\n",
"INFO:tensorflow:Evaluation [9/10]\n",
"INFO:tensorflow:Evaluation [10/10]\n",
"INFO:tensorflow:Finished evaluation at 2018-11-10-03:25:46\n",
"INFO:tensorflow:Saving dict for global step 13000: global_step = 13000, loss = 1.3118937, metrics-translate_enfr_wmt_small32k/targets/accuracy = 0.69891125, metrics-translate_enfr_wmt_small32k/targets/accuracy_per_sequence = 0.09259259, metrics-translate_enfr_wmt_small32k/targets/accuracy_top5 = 0.8764657, metrics-translate_enfr_wmt_small32k/targets/approx_bleu_score = 0.43674964, metrics-translate_enfr_wmt_small32k/targets/neg_log_perplexity = -1.3870174, metrics-translate_enfr_wmt_small32k/targets/rouge_2_fscore = 0.5057337, metrics-translate_enfr_wmt_small32k/targets/rouge_L_fscore = 0.68062705\n",
"INFO:tensorflow:Saving 'checkpoint_path' summary for global step 13000: /root/t2t_train/translate/model.ckpt-13000\n",
"INFO:tensorflow:global_step/sec: 2.99188\n",
"INFO:tensorflow:loss = 1.766869, step = 13000 (33.424 sec)\n",
"INFO:tensorflow:global_step/sec: 4.07759\n",
"INFO:tensorflow:loss = 1.576085, step = 13100 (24.524 sec)\n",
"INFO:tensorflow:global_step/sec: 4.09176\n",
"INFO:tensorflow:loss = 1.7102598, step = 13200 (24.440 sec)\n",
"INFO:tensorflow:global_step/sec: 4.0361\n",
"INFO:tensorflow:loss = 1.7650995, step = 13300 (24.776 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05051\n",
"INFO:tensorflow:loss = 1.8354156, step = 13400 (24.688 sec)\n",
"INFO:tensorflow:global_step/sec: 4.06408\n",
"INFO:tensorflow:loss = 1.5518329, step = 13500 (24.606 sec)\n",
"INFO:tensorflow:global_step/sec: 4.06318\n",
"INFO:tensorflow:loss = 1.7730188, step = 13600 (24.612 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04613\n",
"INFO:tensorflow:loss = 1.7823734, step = 13700 (24.715 sec)\n",
"INFO:tensorflow:global_step/sec: 4.06215\n",
"INFO:tensorflow:loss = 1.572845, step = 13800 (24.617 sec)\n",
"INFO:tensorflow:global_step/sec: 4.02251\n",
"INFO:tensorflow:loss = 1.7843387, step = 13900 (24.860 sec)\n",
"INFO:tensorflow:Saving checkpoints for 14000 into /root/t2t_train/translate/model.ckpt.\n",
"INFO:tensorflow:Skip the current checkpoint eval due to throttle secs (600 secs).\n",
"INFO:tensorflow:global_step/sec: 3.9226\n",
"INFO:tensorflow:loss = 1.5492922, step = 14000 (25.493 sec)\n",
"INFO:tensorflow:global_step/sec: 4.08888\n",
"INFO:tensorflow:loss = 1.6474577, step = 14100 (24.457 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04335\n",
"INFO:tensorflow:loss = 1.7681777, step = 14200 (24.731 sec)\n",
"INFO:tensorflow:global_step/sec: 4.03398\n",
"INFO:tensorflow:loss = 1.4473356, step = 14300 (24.790 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04962\n",
"INFO:tensorflow:loss = 1.6213771, step = 14400 (24.693 sec)\n",
"INFO:tensorflow:global_step/sec: 4.05891\n",
"INFO:tensorflow:loss = 1.6192071, step = 14500 (24.638 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04948\n",
"INFO:tensorflow:loss = 1.5453464, step = 14600 (24.694 sec)\n",
"INFO:tensorflow:global_step/sec: 4.04696\n",
"INFO:tensorflow:loss = 1.6643823, step = 14700 (24.710 sec)\n",
"INFO:tensorflow:global_step/sec: 4.03571\n",
"INFO:tensorflow:loss = 1.6497217, step = 14800 (24.779 sec)\n",
"INFO:tensorflow:global_step/sec: 4.08075\n",
"INFO:tensorflow:loss = 1.6651732, step = 14900 (24.506 sec)\n",
"INFO:tensorflow:Saving checkpoints for 15000 into /root/t2t_train/translate/model.ckpt.\n",
"INFO:tensorflow:Skip the current checkpoint eval due to throttle secs (600 secs).\n",
"INFO:tensorflow:Reading data files from /root/t2t_data/translate_enfr_wmt_small32k-dev*\n",
"INFO:tensorflow:partition: 0 num_data_files: 1\n",
"INFO:tensorflow:Calling model_fn.\n",
"INFO:tensorflow:Setting T2TModel mode to 'eval'\n",
"INFO:tensorflow:Setting hparams.layer_prepostprocess_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.symbol_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.label_smoothing to 0.0\n",
"INFO:tensorflow:Setting hparams.attention_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.relu_dropout to 0.0\n",
":::MLPv0.5.0 transformer 1541820842.865621090 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py:213) model_hp_initializer_gain: 1.0\n",
"INFO:tensorflow:Using variable initializer: uniform_unit_scaling\n",
"INFO:tensorflow:Transforming feature 'inputs' with symbol_modality_27664_256.bottom\n",
"INFO:tensorflow:Transforming feature 'targets' with symbol_modality_27664_256.targets_bottom\n",
"INFO:tensorflow:Building model body\n",
":::MLPv0.5.0 transformer 1541820843.259753942 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:186) model_hp_layer_postprocess_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820843.261769056 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_hidden_layers: 2\n",
":::MLPv0.5.0 transformer 1541820843.264256954 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_attention_num_heads: 4\n",
":::MLPv0.5.0 transformer 1541820843.266128063 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_attention_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820843.571862936 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541820843.573852062 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541820843.575784922 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820844.391736031 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541820844.393747091 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541820844.395627975 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820844.507793903 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_norm: {\"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541820844.634799004 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:202) model_hp_layer_postprocess_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820844.638288021 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_hidden_layers: 2\n",
":::MLPv0.5.0 transformer 1541820844.641011000 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_attention_num_heads: 4\n",
":::MLPv0.5.0 transformer 1541820844.642963886 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_attention_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820845.198302031 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541820845.200350046 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541820845.202279091 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820845.842565060 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541820845.844618082 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541820845.846560955 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820845.932379007 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_norm: {\"hidden_size\": 256}\n",
"INFO:tensorflow:Transforming body output with symbol_modality_27664_256.top\n",
"INFO:tensorflow:Done calling model_fn.\n",
"INFO:tensorflow:Starting evaluation at 2018-11-10-03:34:06\n",
"INFO:tensorflow:Graph was finalized.\n",
"2018-11-10 03:34:07.568385: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0\n",
"2018-11-10 03:34:07.568488: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2018-11-10 03:34:07.568518: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 \n",
"2018-11-10 03:34:07.568540: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N \n",
"2018-11-10 03:34:07.568810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10869 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
"INFO:tensorflow:Restoring parameters from /root/t2t_train/translate/model.ckpt-15000\n",
"INFO:tensorflow:Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"INFO:tensorflow:Evaluation [1/10]\n",
"INFO:tensorflow:Evaluation [2/10]\n",
"INFO:tensorflow:Evaluation [3/10]\n",
"INFO:tensorflow:Evaluation [4/10]\n",
"INFO:tensorflow:Evaluation [5/10]\n",
"INFO:tensorflow:Evaluation [6/10]\n",
"INFO:tensorflow:Evaluation [7/10]\n",
"INFO:tensorflow:Evaluation [8/10]\n",
"INFO:tensorflow:Evaluation [9/10]\n",
"INFO:tensorflow:Evaluation [10/10]\n",
"INFO:tensorflow:Finished evaluation at 2018-11-10-03:34:09\n",
"INFO:tensorflow:Saving dict for global step 15000: global_step = 15000, loss = 1.2668478, metrics-translate_enfr_wmt_small32k/targets/accuracy = 0.70142376, metrics-translate_enfr_wmt_small32k/targets/accuracy_per_sequence = 0.0787037, metrics-translate_enfr_wmt_small32k/targets/accuracy_top5 = 0.8844221, metrics-translate_enfr_wmt_small32k/targets/approx_bleu_score = 0.4403817, metrics-translate_enfr_wmt_small32k/targets/neg_log_perplexity = -1.3417606, metrics-translate_enfr_wmt_small32k/targets/rouge_2_fscore = 0.5115612, metrics-translate_enfr_wmt_small32k/targets/rouge_L_fscore = 0.68328077\n",
"INFO:tensorflow:Saving 'checkpoint_path' summary for global step 15000: /root/t2t_train/translate/model.ckpt-15000\n",
"INFO:tensorflow:Loss for final step: 1.5855087.\n",
"INFO:tensorflow:Reading data files from /root/t2t_data/translate_enfr_wmt_small32k-dev*\n",
"INFO:tensorflow:partition: 0 num_data_files: 1\n",
"INFO:tensorflow:Calling model_fn.\n",
"INFO:tensorflow:Setting T2TModel mode to 'eval'\n",
"INFO:tensorflow:Setting hparams.layer_prepostprocess_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.symbol_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.label_smoothing to 0.0\n",
"INFO:tensorflow:Setting hparams.attention_dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.dropout to 0.0\n",
"INFO:tensorflow:Setting hparams.relu_dropout to 0.0\n",
":::MLPv0.5.0 transformer 1541820850.851739883 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py:213) model_hp_initializer_gain: 1.0\n",
"INFO:tensorflow:Using variable initializer: uniform_unit_scaling\n",
"INFO:tensorflow:Transforming feature 'inputs' with symbol_modality_27664_256.bottom\n",
"INFO:tensorflow:Transforming feature 'targets' with symbol_modality_27664_256.targets_bottom\n",
"INFO:tensorflow:Building model body\n",
":::MLPv0.5.0 transformer 1541820851.221345901 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:186) model_hp_layer_postprocess_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820851.222920895 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_hidden_layers: 2\n",
":::MLPv0.5.0 transformer 1541820851.224296093 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_attention_num_heads: 4\n",
":::MLPv0.5.0 transformer 1541820851.225632906 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_attention_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820851.522963047 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541820851.524425030 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541820851.525876045 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820851.910656929 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541820851.912192106 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541820851.913585901 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/transformer_layers.py:182) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820852.017996073 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:101) model_hp_norm: {\"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541820852.138930082 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:202) model_hp_layer_postprocess_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820852.140409946 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_hidden_layers: 2\n",
":::MLPv0.5.0 transformer 1541820852.141819000 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_attention_num_heads: 4\n",
":::MLPv0.5.0 transformer 1541820852.143171072 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_attention_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820852.681880951 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541820852.683346987 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541820852.684741974 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820853.297072887 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_filter: {\"filter_size\": 1024, \"activation\": \"relu\", \"use_bias\": \"True\"}\n",
":::MLPv0.5.0 transformer 1541820853.298517942 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_ffn_dense: {\"use_bias\": \"True\", \"hidden_size\": 256}\n",
":::MLPv0.5.0 transformer 1541820853.299875021 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:1290) model_hp_relu_dropout: 0.0\n",
":::MLPv0.5.0 transformer 1541820853.382067919 (/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:153) model_hp_norm: {\"hidden_size\": 256}\n",
"INFO:tensorflow:Transforming body output with symbol_modality_27664_256.top\n",
"INFO:tensorflow:Done calling model_fn.\n",
"INFO:tensorflow:Starting evaluation at 2018-11-10-03:34:14\n",
"INFO:tensorflow:Graph was finalized.\n",
"2018-11-10 03:34:14.658547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0\n",
"2018-11-10 03:34:14.658711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2018-11-10 03:34:14.658742: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 \n",
"2018-11-10 03:34:14.658769: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N \n",
"2018-11-10 03:34:14.659059: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10869 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
"INFO:tensorflow:Restoring parameters from /root/t2t_train/translate/model.ckpt-15000\n",
"INFO:tensorflow:Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"INFO:tensorflow:Evaluation [1/10]\n",
"INFO:tensorflow:Evaluation [2/10]\n",
"INFO:tensorflow:Evaluation [3/10]\n",
"INFO:tensorflow:Evaluation [4/10]\n",
"INFO:tensorflow:Evaluation [5/10]\n",
"INFO:tensorflow:Evaluation [6/10]\n",
"INFO:tensorflow:Evaluation [7/10]\n",
"INFO:tensorflow:Evaluation [8/10]\n",
"INFO:tensorflow:Evaluation [9/10]\n",
"INFO:tensorflow:Evaluation [10/10]\n",
"INFO:tensorflow:Finished evaluation at 2018-11-10-03:34:16\n",
"INFO:tensorflow:Saving dict for global step 15000: global_step = 15000, loss = 1.2668478, metrics-translate_enfr_wmt_small32k/targets/accuracy = 0.70142376, metrics-translate_enfr_wmt_small32k/targets/accuracy_per_sequence = 0.0787037, metrics-translate_enfr_wmt_small32k/targets/accuracy_top5 = 0.8844221, metrics-translate_enfr_wmt_small32k/targets/approx_bleu_score = 0.4403817, metrics-translate_enfr_wmt_small32k/targets/neg_log_perplexity = -1.3417606, metrics-translate_enfr_wmt_small32k/targets/rouge_2_fscore = 0.5115612, metrics-translate_enfr_wmt_small32k/targets/rouge_L_fscore = 0.68328077\n",
"INFO:tensorflow:Saving 'checkpoint_path' summary for global step 15000: /root/t2t_train/translate/model.ckpt-15000\n",
":::MLPv0.5.0 transformer 1541820856.729504108 (/usr/local/bin/t2t-trainer:28) run_final\n"
],
"name": "stdout"
}
]
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment