Last active
December 3, 2018 07:59
-
-
Save aleksas/a8182011193498d70a09172cd5baa8d3 to your computer and use it in GitHub Desktop.
Run imdb sentiment analysis
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| PROBLEM=sentiment_imdb_characters | |
| MODEL=transformer_encoder | |
| HPARAMS=transformer_base_single_gpu | |
| DATA_DIR=$HOME/t2t_data | |
| TMP_DIR=/tmp/t2t_datagen | |
| TRAIN_DIR=$HOME/t2t_train/$PROBLEM/$MODEL-$HPARAMS | |
| BATCH_SIZE=512 | |
| mkdir -p $DATA_DIR $TMP_DIR | |
| t2t-datagen \ | |
| --data_dir=$DATA_DIR \ | |
| --tmp_dir=$TMP_DIR \ | |
| --problem=$PROBLEM | |
| t2t-trainer \ | |
| --data_dir=$DATA_DIR \ | |
| --problem=$PROBLEM \ | |
| --model=$MODEL \ | |
| --hparams_set=$HPARAMS \ | |
| --output_dir=$TRAIN_DIR \ | |
| --tmp_dir=$TMP_DIR \ | |
| --batch_size=$BATCH_SIZE \ | |
| --worker_gpu=$WORKER_GPU |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| INFO:tensorflow:Found unparsed command-line arguments. Checking if any start with --hp_ and interpreting those as hparams settings. | |
| WARNING:tensorflow:Found unknown flag: --batch_size=512 | |
| WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/trainer_lib.py:165: __init__ (from tensorflow.contrib.learn.python.learn.estimators.run_config) is deprecated and will be removed in a future version. | |
| Instructions for updating: | |
| When switching to tf.estimator.Estimator, use tf.estimator.RunConfig instead. | |
| INFO:tensorflow:schedule=continuous_train_and_eval | |
| INFO:tensorflow:worker_gpu=1 | |
| INFO:tensorflow:sync=False | |
| WARNING:tensorflow:Schedule=continuous_train_and_eval. Assuming that training is running on a single machine. | |
| INFO:tensorflow:datashard_devices: ['gpu:0'] | |
| INFO:tensorflow:caching_devices: None | |
| INFO:tensorflow:ps_devices: ['gpu:0'] | |
| INFO:tensorflow:Using config: {'_save_checkpoints_secs': None, '_keep_checkpoint_max': 20, '_task_type': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fcafc933110>, '_keep_checkpoint_every_n_hours': 10000, '_session_config': gpu_options { | |
| per_process_gpu_memory_fraction: 0.95 | |
| } | |
| allow_soft_placement: true | |
| graph_options { | |
| optimizer_options { | |
| } | |
| } | |
| , 'use_tpu': False, '_tf_random_seed': None, '_num_worker_replicas': 0, '_task_id': 0, 't2t_device_info': {'num_async_replicas': 1}, '_evaluation_master': '', '_log_step_count_steps': 100, '_num_ps_replicas': 0, '_train_distribute': None, '_is_chief': True, '_tf_config': gpu_options { | |
| per_process_gpu_memory_fraction: 1.0 | |
| } | |
| , '_save_checkpoints_steps': 1000, '_environment': 'local', '_master': '', '_model_dir': '/root/t2t_train/sentiment_imdb_characters/transformer_encoder-transformer_base', 'data_parallelism': <tensor2tensor.utils.expert_utils.Parallelism object at 0x7fcafc933150>, '_save_summary_steps': 100} | |
| WARNING:tensorflow:Estimator's model_fn (<function wrapping_model_fn at 0x7fcafc8c7c80>) includes params argument, but params are not passed to Estimator. | |
| WARNING:tensorflow:ValidationMonitor only works with --schedule=train_and_evaluate | |
| INFO:tensorflow:Running training and evaluation locally (non-distributed). | |
| INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after 600 secs (eval_spec.throttle_secs) or training is finished. | |
| INFO:tensorflow:Reading data files from /root/t2t_data/sentiment_imdb_characters-train* | |
| INFO:tensorflow:partition: 0 num_data_files: 10 | |
| INFO:tensorflow:Calling model_fn. | |
| INFO:tensorflow:Unsetting shared_embedding_and_softmax_weights. | |
| INFO:tensorflow:Setting T2TModel mode to 'train' | |
| INFO:tensorflow:Using variable initializer: uniform_unit_scaling | |
| INFO:tensorflow:Transforming feature 'inputs' with symbol_modality_258_512.bottom | |
| INFO:tensorflow:Transforming 'targets' with class_label_modality_2_512.targets_bottom | |
| INFO:tensorflow:Building model body | |
| INFO:tensorflow:Transforming body output with class_label_modality_2_512.top | |
| WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/modalities.py:703: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. | |
| Instructions for updating: | |
| keep_dims is deprecated, use keepdims instead | |
| INFO:tensorflow:Base learning rate: 2.000000 | |
| INFO:tensorflow:Trainable Variables Total size: 19052546 | |
| INFO:tensorflow:Using optimizer Adam | |
| /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gradients_impl.py:100: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. | |
| "Converting sparse IndexedSlices to a dense Tensor of unknown shape. " | |
| INFO:tensorflow:Done calling model_fn. | |
| INFO:tensorflow:Create CheckpointSaverHook. | |
| INFO:tensorflow:Graph was finalized. | |
| 2018-12-03 00:32:43.738552: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA | |
| 2018-12-03 00:32:43.800682: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero | |
| 2018-12-03 00:32:43.801476: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties: | |
| name: GeForce GTX 1070 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.683 | |
| pciBusID: 0000:01:00.0 | |
| totalMemory: 7.93GiB freeMemory: 7.81GiB | |
| 2018-12-03 00:32:43.878655: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero | |
| 2018-12-03 00:32:43.879407: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 1 with properties: | |
| name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.835 | |
| pciBusID: 0000:07:00.0 | |
| totalMemory: 7.93GiB freeMemory: 7.84GiB | |
| 2018-12-03 00:32:43.880314: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0, 1 | |
| 2018-12-03 00:32:44.242779: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix: | |
| 2018-12-03 00:32:44.242815: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0 1 | |
| 2018-12-03 00:32:44.242822: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N Y | |
| 2018-12-03 00:32:44.242826: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 1: Y N | |
| 2018-12-03 00:32:44.243341: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7712 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1) | |
| 2018-12-03 00:32:44.315487: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 7713 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1070, pci bus id: 0000:07:00.0, compute capability: 6.1) | |
| INFO:tensorflow:Restoring parameters from /root/t2t_train/sentiment_imdb_characters/transformer_encoder-transformer_base/model.ckpt-15767 | |
| INFO:tensorflow:Running local_init_op. | |
| INFO:tensorflow:Done running local_init_op. | |
| INFO:tensorflow:Saving checkpoints for 15768 into /root/t2t_train/sentiment_imdb_characters/transformer_encoder-transformer_base/model.ckpt. | |
| INFO:tensorflow:loss = 0.07068637, step = 15767 | |
| INFO:tensorflow:global_step/sec: 7.91213 | |
| INFO:tensorflow:loss = 0.22584555, step = 15867 (12.640 sec) | |
| INFO:tensorflow:global_step/sec: 9.96565 | |
| INFO:tensorflow:loss = 0.10638794, step = 15967 (10.034 sec) | |
| INFO:tensorflow:global_step/sec: 9.9829 | |
| INFO:tensorflow:loss = 0.22673553, step = 16067 (10.017 sec) | |
| INFO:tensorflow:global_step/sec: 10.0112 | |
| INFO:tensorflow:loss = 0.08368318, step = 16167 (9.989 sec) | |
| INFO:tensorflow:global_step/sec: 10.0217 | |
| INFO:tensorflow:loss = 0.215715, step = 16267 (9.978 sec) | |
| INFO:tensorflow:global_step/sec: 9.93835 | |
| INFO:tensorflow:loss = 0.19894557, step = 16367 (10.062 sec) | |
| INFO:tensorflow:global_step/sec: 9.94434 | |
| INFO:tensorflow:loss = 0.16500695, step = 16467 (10.056 sec) | |
| INFO:tensorflow:global_step/sec: 9.97028 | |
| INFO:tensorflow:loss = 0.09865117, step = 16567 (10.030 sec) | |
| INFO:tensorflow:global_step/sec: 9.95326 | |
| INFO:tensorflow:loss = 0.26081014, step = 16667 (10.047 sec) | |
| INFO:tensorflow:Saving checkpoints for 16768 into /root/t2t_train/sentiment_imdb_characters/transformer_encoder-transformer_base/model.ckpt. | |
| INFO:tensorflow:global_step/sec: 9.30917 | |
| INFO:tensorflow:loss = 0.29037058, step = 16767 (10.742 sec) | |
| INFO:tensorflow:global_step/sec: 9.94707 | |
| INFO:tensorflow:loss = 0.12956437, step = 16867 (10.053 sec) | |
| INFO:tensorflow:global_step/sec: 9.94971 | |
| INFO:tensorflow:loss = 0.16136491, step = 16967 (10.051 sec) | |
| INFO:tensorflow:global_step/sec: 9.91337 | |
| INFO:tensorflow:loss = 0.12763578, step = 17067 (10.088 sec) | |
| INFO:tensorflow:global_step/sec: 9.97755 | |
| INFO:tensorflow:loss = 0.07679598, step = 17167 (10.022 sec) | |
| INFO:tensorflow:global_step/sec: 9.96532 | |
| INFO:tensorflow:loss = 0.21029404, step = 17267 (10.035 sec) | |
| INFO:tensorflow:global_step/sec: 9.95546 | |
| INFO:tensorflow:loss = 0.14709087, step = 17367 (10.045 sec) | |
| INFO:tensorflow:global_step/sec: 9.98741 | |
| INFO:tensorflow:loss = 0.19409588, step = 17467 (10.013 sec) | |
| INFO:tensorflow:global_step/sec: 9.92126 | |
| INFO:tensorflow:loss = 0.050190523, step = 17567 (10.079 sec) | |
| INFO:tensorflow:global_step/sec: 9.94801 | |
| INFO:tensorflow:loss = 0.081334405, step = 17667 (10.052 sec) | |
| INFO:tensorflow:Saving checkpoints for 17768 into /root/t2t_train/sentiment_imdb_characters/transformer_encoder-transformer_base/model.ckpt. | |
| INFO:tensorflow:global_step/sec: 9.10562 | |
| INFO:tensorflow:loss = 0.038878243, step = 17767 (10.983 sec) | |
| INFO:tensorflow:global_step/sec: 9.92217 | |
| INFO:tensorflow:loss = 0.038238686, step = 17867 (10.078 sec) | |
| INFO:tensorflow:global_step/sec: 9.90845 | |
| INFO:tensorflow:loss = 0.083561644, step = 17967 (10.092 sec) | |
| INFO:tensorflow:global_step/sec: 9.94442 | |
| INFO:tensorflow:loss = 0.109358564, step = 18067 (10.056 sec) | |
| INFO:tensorflow:global_step/sec: 9.91609 | |
| INFO:tensorflow:loss = 0.09136684, step = 18167 (10.085 sec) | |
| INFO:tensorflow:global_step/sec: 9.90882 | |
| INFO:tensorflow:loss = 0.09995653, step = 18267 (10.095 sec) | |
| INFO:tensorflow:global_step/sec: 9.90739 | |
| INFO:tensorflow:loss = 0.12213722, step = 18367 (10.090 sec) | |
| INFO:tensorflow:global_step/sec: 9.9119 | |
| INFO:tensorflow:loss = 0.09872891, step = 18467 (10.089 sec) | |
| INFO:tensorflow:global_step/sec: 9.9951 | |
| INFO:tensorflow:loss = 0.088943064, step = 18567 (10.005 sec) | |
| INFO:tensorflow:global_step/sec: 9.89823 | |
| INFO:tensorflow:loss = 0.20490518, step = 18667 (10.103 sec) | |
| INFO:tensorflow:Saving checkpoints for 18768 into /root/t2t_train/sentiment_imdb_characters/transformer_encoder-transformer_base/model.ckpt. | |
| INFO:tensorflow:global_step/sec: 9.25783 | |
| INFO:tensorflow:loss = 0.08762752, step = 18767 (10.802 sec) | |
| INFO:tensorflow:global_step/sec: 9.90454 | |
| INFO:tensorflow:loss = 0.023553476, step = 18867 (10.096 sec) | |
| INFO:tensorflow:global_step/sec: 9.90736 | |
| INFO:tensorflow:loss = 0.13714536, step = 18967 (10.093 sec) | |
| INFO:tensorflow:global_step/sec: 9.92325 | |
| INFO:tensorflow:loss = 0.0923896, step = 19067 (10.077 sec) | |
| INFO:tensorflow:global_step/sec: 9.96013 | |
| INFO:tensorflow:loss = 0.19316633, step = 19167 (10.040 sec) | |
| INFO:tensorflow:global_step/sec: 9.92856 | |
| INFO:tensorflow:loss = 0.2654686, step = 19267 (10.072 sec) | |
| INFO:tensorflow:global_step/sec: 9.90508 | |
| INFO:tensorflow:loss = 0.12172982, step = 19367 (10.096 sec) | |
| INFO:tensorflow:global_step/sec: 9.88051 | |
| INFO:tensorflow:loss = 0.081762746, step = 19467 (10.122 sec) | |
| INFO:tensorflow:global_step/sec: 9.94692 | |
| INFO:tensorflow:loss = 0.049419664, step = 19567 (10.053 sec) | |
| INFO:tensorflow:global_step/sec: 9.92598 | |
| INFO:tensorflow:loss = 0.25150484, step = 19667 (10.075 sec) | |
| INFO:tensorflow:Saving checkpoints for 19768 into /root/t2t_train/sentiment_imdb_characters/transformer_encoder-transformer_base/model.ckpt. | |
| INFO:tensorflow:global_step/sec: 9.10779 | |
| INFO:tensorflow:loss = 0.04782917, step = 19767 (10.980 sec) | |
| INFO:tensorflow:global_step/sec: 9.89414 | |
| INFO:tensorflow:loss = 0.09455118, step = 19867 (10.107 sec) | |
| INFO:tensorflow:global_step/sec: 10.0053 | |
| INFO:tensorflow:loss = 0.12819602, step = 19967 (9.995 sec) | |
| INFO:tensorflow:global_step/sec: 9.90394 | |
| INFO:tensorflow:loss = 0.10249798, step = 20067 (10.097 sec) | |
| INFO:tensorflow:global_step/sec: 9.90123 | |
| INFO:tensorflow:loss = 0.08660681, step = 20167 (10.100 sec) | |
| INFO:tensorflow:global_step/sec: 9.9449 | |
| INFO:tensorflow:loss = 0.23311824, step = 20267 (10.055 sec) | |
| INFO:tensorflow:global_step/sec: 9.91927 | |
| INFO:tensorflow:loss = 0.08787894, step = 20367 (10.082 sec) | |
| INFO:tensorflow:global_step/sec: 9.92413 | |
| INFO:tensorflow:loss = 0.09562593, step = 20467 (10.076 sec) | |
| INFO:tensorflow:global_step/sec: 9.89343 | |
| INFO:tensorflow:loss = 0.11091391, step = 20567 (10.108 sec) | |
| INFO:tensorflow:global_step/sec: 9.90341 | |
| INFO:tensorflow:loss = 0.037016656, step = 20667 (10.098 sec) | |
| INFO:tensorflow:Saving checkpoints for 20768 into /root/t2t_train/sentiment_imdb_characters/transformer_encoder-transformer_base/model.ckpt. | |
| INFO:tensorflow:global_step/sec: 9.24064 | |
| INFO:tensorflow:loss = 0.049457937, step = 20767 (10.821 sec) | |
| INFO:tensorflow:global_step/sec: 9.9367 | |
| INFO:tensorflow:loss = 0.060066514, step = 20867 (10.063 sec) | |
| INFO:tensorflow:global_step/sec: 9.91909 | |
| INFO:tensorflow:loss = 0.35424185, step = 20967 (10.082 sec) | |
| INFO:tensorflow:global_step/sec: 9.91975 | |
| INFO:tensorflow:loss = 0.20744851, step = 21067 (10.081 sec) | |
| INFO:tensorflow:global_step/sec: 9.90214 | |
| INFO:tensorflow:loss = 0.12818135, step = 21167 (10.099 sec) | |
| INFO:tensorflow:global_step/sec: 9.90567 | |
| INFO:tensorflow:loss = 0.11052539, step = 21267 (10.095 sec) | |
| INFO:tensorflow:global_step/sec: 9.99491 | |
| INFO:tensorflow:loss = 0.045732643, step = 21367 (10.005 sec) | |
| INFO:tensorflow:global_step/sec: 9.88525 | |
| INFO:tensorflow:loss = 0.17503393, step = 21467 (10.116 sec) | |
| INFO:tensorflow:Saving checkpoints for 21558 into /root/t2t_train/sentiment_imdb_characters/transformer_encoder-transformer_base/model.ckpt. | |
| INFO:tensorflow:Loss for final step: 0.15184914. | |
| INFO:tensorflow:Reading data files from /root/t2t_data/sentiment_imdb_characters-dev* | |
| INFO:tensorflow:partition: 0 num_data_files: 1 | |
| INFO:tensorflow:Calling model_fn. | |
| INFO:tensorflow:Unsetting shared_embedding_and_softmax_weights. | |
| INFO:tensorflow:Setting T2TModel mode to 'eval' | |
| INFO:tensorflow:Setting hparams.layer_prepostprocess_dropout to 0.0 | |
| INFO:tensorflow:Setting hparams.symbol_dropout to 0.0 | |
| INFO:tensorflow:Setting hparams.attention_dropout to 0.0 | |
| INFO:tensorflow:Setting hparams.dropout to 0.0 | |
| INFO:tensorflow:Setting hparams.relu_dropout to 0.0 | |
| INFO:tensorflow:Using variable initializer: uniform_unit_scaling | |
| INFO:tensorflow:Transforming feature 'inputs' with symbol_modality_258_512.bottom | |
| INFO:tensorflow:Transforming 'targets' with class_label_modality_2_512.targets_bottom | |
| INFO:tensorflow:Building model body | |
| INFO:tensorflow:Transforming body output with class_label_modality_2_512.top | |
| INFO:tensorflow:Done calling model_fn. | |
| INFO:tensorflow:Starting evaluation at 2018-12-03-00:42:47 | |
| INFO:tensorflow:Graph was finalized. | |
| 2018-12-03 00:42:47.628740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0, 1 | |
| 2018-12-03 00:42:47.628871: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix: | |
| 2018-12-03 00:42:47.628896: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0 1 | |
| 2018-12-03 00:42:47.628904: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N Y | |
| 2018-12-03 00:42:47.628910: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 1: Y N | |
| 2018-12-03 00:42:47.629281: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7712 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1) | |
| 2018-12-03 00:42:47.629472: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 7713 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1070, pci bus id: 0000:07:00.0, compute capability: 6.1) | |
| INFO:tensorflow:Restoring parameters from /root/t2t_train/sentiment_imdb_characters/transformer_encoder-transformer_base/model.ckpt-21558 | |
| INFO:tensorflow:Running local_init_op. | |
| INFO:tensorflow:Done running local_init_op. | |
| 2018-12-03 00:43:03.345590: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.87GiB. Current allocation summary follows. | |
| 2018-12-03 00:43:03.345700: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (256): Total Chunks: 76, Chunks in use: 76. 19.0KiB allocated for chunks. 19.0KiB in use in bin. 458B client-requested in use in bin. | |
| 2018-12-03 00:43:03.345738: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (512): Total Chunks: 1, Chunks in use: 0. 512B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
| 2018-12-03 00:43:03.345768: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1024): Total Chunks: 3, Chunks in use: 3. 3.5KiB allocated for chunks. 3.5KiB in use in bin. 3.0KiB client-requested in use in bin. | |
| 2018-12-03 00:43:03.345796: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2048): Total Chunks: 33, Chunks in use: 32. 66.0KiB allocated for chunks. 64.0KiB in use in bin. 64.0KiB client-requested in use in bin. | |
| 2018-12-03 00:43:03.345825: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4096): Total Chunks: 1, Chunks in use: 1. 4.0KiB allocated for chunks. 4.0KiB in use in bin. 4.0KiB client-requested in use in bin. | |
| 2018-12-03 00:43:03.345880: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8192): Total Chunks: 7, Chunks in use: 6. 68.0KiB allocated for chunks. 57.8KiB in use in bin. 48.0KiB client-requested in use in bin. | |
| 2018-12-03 00:43:03.346005: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
| 2018-12-03 00:43:03.346045: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (32768): Total Chunks: 17, Chunks in use: 16. 548.0KiB allocated for chunks. 516.0KiB in use in bin. 516.0KiB client-requested in use in bin. | |
| 2018-12-03 00:43:03.346084: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (65536): Total Chunks: 4, Chunks in use: 3. 312.5KiB allocated for chunks. 242.5KiB in use in bin. 213.7KiB client-requested in use in bin. | |
| 2018-12-03 00:43:03.346110: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
| 2018-12-03 00:43:03.346133: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
| 2018-12-03 00:43:03.346159: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (524288): Total Chunks: 1, Chunks in use: 0. 915.5KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
| 2018-12-03 00:43:03.346188: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1048576): Total Chunks: 24, Chunks in use: 24. 27.66MiB allocated for chunks. 27.66MiB in use in bin. 24.00MiB client-requested in use in bin. | |
| 2018-12-03 00:43:03.346213: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
| 2018-12-03 00:43:03.346245: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4194304): Total Chunks: 14, Chunks in use: 12. 57.89MiB allocated for chunks. 48.01MiB in use in bin. 48.00MiB client-requested in use in bin. | |
| 2018-12-03 00:43:03.346270: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
| 2018-12-03 00:43:03.346299: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
| 2018-12-03 00:43:03.346335: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (33554432): Total Chunks: 4, Chunks in use: 2. 216.69MiB allocated for chunks. 108.34MiB in use in bin. 108.34MiB client-requested in use in bin. | |
| 2018-12-03 00:43:03.346364: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
| 2018-12-03 00:43:03.346392: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
| 2018-12-03 00:43:03.346420: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (268435456): Total Chunks: 3, Chunks in use: 2. 7.23GiB allocated for chunks. 5.73GiB in use in bin. 5.73GiB client-requested in use in bin. | |
| 2018-12-03 00:43:03.346450: I tensorflow/core/common_runtime/bfc_allocator.cc:646] Bin for 2.87GiB was 256.00MiB, Chunk State: | |
| 2018-12-03 00:43:03.346487: I tensorflow/core/common_runtime/bfc_allocator.cc:652] Size: 1.50GiB | Requested Size: 512B | in_use: 0, prev: Size: 2.87GiB | Requested Size: 2.87GiB | in_use: 1 | |
| 2018-12-03 00:43:03.346514: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0000000 of size 1280 | |
| 2018-12-03 00:43:03.346537: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0000500 of size 1280 | |
| 2018-12-03 00:43:03.346556: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0000a00 of size 256 | |
| 2018-12-03 00:43:03.346586: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0000b00 of size 256 | |
| 2018-12-03 00:43:03.346605: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0000c00 of size 2048 | |
| 2018-12-03 00:43:03.346624: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0001400 of size 8192 | |
| 2018-12-03 00:43:03.346641: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0003400 of size 2048 | |
| 2018-12-03 00:43:03.346660: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0003c00 of size 2048 | |
| 2018-12-03 00:43:03.346679: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0004400 of size 2048 | |
| 2018-12-03 00:43:03.347043: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0004c00 of size 2048 | |
| 2018-12-03 00:43:03.347094: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0005400 of size 2048 | |
| 2018-12-03 00:43:03.347118: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0005c00 of size 12288 | |
| 2018-12-03 00:43:03.347140: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0008c00 of size 2048 | |
| 2018-12-03 00:43:03.347163: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0009400 of size 2048 | |
| 2018-12-03 00:43:03.347186: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0009c00 of size 4096 | |
| 2018-12-03 00:43:03.347208: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a000ac00 of size 32768 | |
| 2018-12-03 00:43:03.347229: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0012c00 of size 32768 | |
| 2018-12-03 00:43:03.347252: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a001ac00 of size 1050624 | |
| 2018-12-03 00:43:03.347274: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a011b400 of size 2048 | |
| 2018-12-03 00:43:03.347296: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a011bc00 of size 2048 | |
| 2018-12-03 00:43:03.347317: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a011c400 of size 2048 | |
| 2018-12-03 00:43:03.347339: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a011cc00 of size 2048 | |
| 2018-12-03 00:43:03.347362: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a011d400 of size 1081344 | |
| 2018-12-03 00:43:03.347384: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0225400 of size 32768 | |
| 2018-12-03 00:43:03.347403: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a022d400 of size 1955840 | |
| 2018-12-03 00:43:03.347420: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a040ac00 of size 2048 | |
| 2018-12-03 00:43:03.347438: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a040b400 of size 2048 | |
| 2018-12-03 00:43:03.347457: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a040bc00 of size 71680 | |
| 2018-12-03 00:43:03.347483: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a041d400 of size 1048576 | |
| 2018-12-03 00:43:03.347502: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a051d400 of size 1048576 | |
| 2018-12-03 00:43:03.347521: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a061d400 of size 2056192 | |
| 2018-12-03 00:43:03.347539: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0813400 of size 2048 | |
| 2018-12-03 00:43:03.347556: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0813c00 of size 2048 | |
| 2018-12-03 00:43:03.347574: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0814400 of size 32768 | |
| 2018-12-03 00:43:03.347595: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a081c400 of size 2048 | |
| 2018-12-03 00:43:03.347613: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a081cc00 of size 2048 | |
| 2018-12-03 00:43:03.347631: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a081d400 of size 2048 | |
| 2018-12-03 00:43:03.347651: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a081dc00 of size 2048 | |
| 2018-12-03 00:43:03.347672: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a081e400 of size 2048 | |
| 2018-12-03 00:43:03.347699: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a081ec00 of size 2048 | |
| 2018-12-03 00:43:03.347721: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a081f400 of size 2048 | |
| 2018-12-03 00:43:03.347740: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a081fc00 of size 2048 | |
| 2018-12-03 00:43:03.347757: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0820400 of size 2048 | |
| 2018-12-03 00:43:03.347776: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0820c00 of size 2048 | |
| 2018-12-03 00:43:03.347798: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0821400 of size 10240 | |
| 2018-12-03 00:43:03.347820: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0823c00 of size 256 | |
| 2018-12-03 00:43:03.347841: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0823d00 of size 256 | |
| 2018-12-03 00:43:03.347863: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0823e00 of size 256 | |
| 2018-12-03 00:43:03.347884: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0823f00 of size 256 | |
| 2018-12-03 00:43:03.347905: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0824000 of size 256 | |
| 2018-12-03 00:43:03.347926: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0824100 of size 256 | |
| 2018-12-03 00:43:03.347948: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0824200 of size 256 | |
| 2018-12-03 00:43:03.347970: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0824300 of size 256 | |
| 2018-12-03 00:43:03.347990: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0824400 of size 2048 | |
| 2018-12-03 00:43:03.348011: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0824c00 of size 2048 | |
| 2018-12-03 00:43:03.348033: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fc8a0825400 of size 71680 | |
| 2018-12-03 00:43:03.348054: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0836c00 of size 32768 | |
| 2018-12-03 00:43:03.348077: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a083ec00 of size 1990656 | |
| 2018-12-03 00:43:03.348100: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0a24c00 of size 1048576 | |
| 2018-12-03 00:43:03.348122: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0b24c00 of size 1048576 | |
| 2018-12-03 00:43:03.348143: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0c24c00 of size 8192 | |
| 2018-12-03 00:43:03.348169: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0c26c00 of size 8192 | |
| 2018-12-03 00:43:03.348188: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0c28c00 of size 256 | |
| 2018-12-03 00:43:03.348209: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0c28d00 of size 2048 | |
| 2018-12-03 00:43:03.348266: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0c29500 of size 2048 | |
| 2018-12-03 00:43:03.348287: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0c29d00 of size 12032 | |
| 2018-12-03 00:43:03.348306: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0c2cc00 of size 4194304 | |
| 2018-12-03 00:43:03.348327: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a102cc00 of size 32768 | |
| 2018-12-03 00:43:03.348348: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a1034c00 of size 32768 | |
| 2018-12-03 00:43:03.348367: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a103cc00 of size 1048576 | |
| 2018-12-03 00:43:03.348388: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a113cc00 of size 32768 | |
| 2018-12-03 00:43:03.348406: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a1144c00 of size 1048576 | |
| 2018-12-03 00:43:03.348427: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a1244c00 of size 1048576 | |
| 2018-12-03 00:43:03.348449: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a1344c00 of size 32768 | |
| 2018-12-03 00:43:03.348468: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a134cc00 of size 1966080 | |
| 2018-12-03 00:43:03.348487: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a152cc00 of size 1081344 | |
| 2018-12-03 00:43:03.348511: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a1634c00 of size 32768 | |
| 2018-12-03 00:43:03.348533: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a163cc00 of size 4194304 | |
| 2018-12-03 00:43:03.348555: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a1a3cc00 of size 4194304 | |
| 2018-12-03 00:43:03.348575: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a1e3cc00 of size 32768 | |
| 2018-12-03 00:43:03.348593: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a1e44c00 of size 4194304 | |
| 2018-12-03 00:43:03.348611: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2244c00 of size 34816 | |
| 2018-12-03 00:43:03.348629: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a224d400 of size 32768 | |
| 2018-12-03 00:43:03.348653: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2255400 of size 34816 | |
| 2018-12-03 00:43:03.348675: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a225dc00 of size 65536 | |
| 2018-12-03 00:43:03.348698: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a226dc00 of size 1048576 | |
| 2018-12-03 00:43:03.348716: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a236dc00 of size 4194304 | |
| 2018-12-03 00:43:03.348735: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fc8a276dc00 of size 6166528 | |
| 2018-12-03 00:43:03.348754: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d4f400 of size 32768 | |
| 2018-12-03 00:43:03.348772: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d57400 of size 2048 | |
| 2018-12-03 00:43:03.348793: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d57c00 of size 2048 | |
| 2018-12-03 00:43:03.348812: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58400 of size 256 | |
| 2018-12-03 00:43:03.348829: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58500 of size 256 | |
| 2018-12-03 00:43:03.348847: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58600 of size 256 | |
| 2018-12-03 00:43:03.348865: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58700 of size 256 | |
| 2018-12-03 00:43:03.348886: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58800 of size 256 | |
| 2018-12-03 00:43:03.348906: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58900 of size 256 | |
| 2018-12-03 00:43:03.348927: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58a00 of size 256 | |
| 2018-12-03 00:43:03.348946: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58b00 of size 256 | |
| 2018-12-03 00:43:03.348964: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58c00 of size 256 | |
| 2018-12-03 00:43:03.348982: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58d00 of size 256 | |
| 2018-12-03 00:43:03.348999: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58e00 of size 256 | |
| 2018-12-03 00:43:03.349017: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58f00 of size 256 | |
| 2018-12-03 00:43:03.349034: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59000 of size 256 | |
| 2018-12-03 00:43:03.349052: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59100 of size 256 | |
| 2018-12-03 00:43:03.349069: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59200 of size 256 | |
| 2018-12-03 00:43:03.349087: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59300 of size 256 | |
| 2018-12-03 00:43:03.349112: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59400 of size 256 | |
| 2018-12-03 00:43:03.349133: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59500 of size 256 | |
| 2018-12-03 00:43:03.349154: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59600 of size 256 | |
| 2018-12-03 00:43:03.349179: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59700 of size 256 | |
| 2018-12-03 00:43:03.349200: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59800 of size 256 | |
| 2018-12-03 00:43:03.349221: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59900 of size 256 | |
| 2018-12-03 00:43:03.349243: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59a00 of size 256 | |
| 2018-12-03 00:43:03.349263: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59b00 of size 256 | |
| 2018-12-03 00:43:03.349284: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59c00 of size 256 | |
| 2018-12-03 00:43:03.349308: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59d00 of size 256 | |
| 2018-12-03 00:43:03.349327: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59e00 of size 256 | |
| 2018-12-03 00:43:03.349347: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59f00 of size 256 | |
| 2018-12-03 00:43:03.349369: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5a000 of size 256 | |
| 2018-12-03 00:43:03.349389: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5a100 of size 256 | |
| 2018-12-03 00:43:03.349410: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5a200 of size 256 | |
| 2018-12-03 00:43:03.349432: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5a300 of size 256 | |
| 2018-12-03 00:43:03.349453: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5a400 of size 256 | |
| 2018-12-03 00:43:03.349476: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5a500 of size 256 | |
| 2018-12-03 00:43:03.349494: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5a600 of size 256 | |
| 2018-12-03 00:43:03.349511: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5a700 of size 256 | |
| 2018-12-03 00:43:03.349532: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5a800 of size 256 | |
| 2018-12-03 00:43:03.349552: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5a900 of size 256 | |
| 2018-12-03 00:43:03.349573: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5aa00 of size 256 | |
| 2018-12-03 00:43:03.349593: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5ab00 of size 256 | |
| 2018-12-03 00:43:03.349613: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5ac00 of size 256 | |
| 2018-12-03 00:43:03.349634: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5ad00 of size 256 | |
| 2018-12-03 00:43:03.349654: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5ae00 of size 256 | |
| 2018-12-03 00:43:03.349674: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5af00 of size 256 | |
| 2018-12-03 00:43:03.349695: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5b000 of size 256 | |
| 2018-12-03 00:43:03.349715: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5b100 of size 256 | |
| 2018-12-03 00:43:03.349735: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5b200 of size 256 | |
| 2018-12-03 00:43:03.349756: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5b300 of size 256 | |
| 2018-12-03 00:43:03.349777: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5b400 of size 256 | |
| 2018-12-03 00:43:03.349797: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5b500 of size 256 | |
| 2018-12-03 00:43:03.349817: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5b600 of size 256 | |
| 2018-12-03 00:43:03.349838: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5b700 of size 256 | |
| 2018-12-03 00:43:03.349858: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5b800 of size 256 | |
| 2018-12-03 00:43:03.349880: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5b900 of size 1024 | |
| 2018-12-03 00:43:03.349899: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5bd00 of size 256 | |
| 2018-12-03 00:43:03.349917: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5be00 of size 256 | |
| 2018-12-03 00:43:03.349938: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5bf00 of size 256 | |
| 2018-12-03 00:43:03.349958: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5c000 of size 256 | |
| 2018-12-03 00:43:03.349978: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5c100 of size 256 | |
| 2018-12-03 00:43:03.349998: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5c200 of size 256 | |
| 2018-12-03 00:43:03.350018: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5c300 of size 256 | |
| 2018-12-03 00:43:03.350039: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5c400 of size 256 | |
| 2018-12-03 00:43:03.350059: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fc8a2d5c500 of size 2048 | |
| 2018-12-03 00:43:03.350080: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5cd00 of size 256 | |
| 2018-12-03 00:43:03.350101: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5ce00 of size 256 | |
| 2018-12-03 00:43:03.350121: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5cf00 of size 256 | |
| 2018-12-03 00:43:03.350141: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fc8a2d5d000 of size 512 | |
| 2018-12-03 00:43:03.350161: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5d200 of size 256 | |
| 2018-12-03 00:43:03.350185: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fc8a2d5d300 of size 10496 | |
| 2018-12-03 00:43:03.350203: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5fc00 of size 32768 | |
| 2018-12-03 00:43:03.350222: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fc8a2d67c00 of size 32768 | |
| 2018-12-03 00:43:03.350239: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d6fc00 of size 4194304 | |
| 2018-12-03 00:43:03.350260: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a316fc00 of size 1048576 | |
| 2018-12-03 00:43:03.350280: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a326fc00 of size 1048576 | |
| 2018-12-03 00:43:03.350301: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a336fc00 of size 4194304 | |
| 2018-12-03 00:43:03.350321: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a376fc00 of size 1048576 | |
| 2018-12-03 00:43:03.350342: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a386fc00 of size 1048576 | |
| 2018-12-03 00:43:03.350362: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a396fc00 of size 4194304 | |
| 2018-12-03 00:43:03.350382: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a3d6fc00 of size 1048576 | |
| 2018-12-03 00:43:03.350404: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a3e6fc00 of size 1048576 | |
| 2018-12-03 00:43:03.350426: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a3f6fc00 of size 1048576 | |
| 2018-12-03 00:43:03.350446: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a406fc00 of size 1048576 | |
| 2018-12-03 00:43:03.350465: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a416fc00 of size 111104 | |
| 2018-12-03 00:43:03.350482: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fc8a418ae00 of size 937472 | |
| 2018-12-03 00:43:03.350503: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a426fc00 of size 1048576 | |
| 2018-12-03 00:43:03.350524: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a436fc00 of size 4202496 | |
| 2018-12-03 00:43:03.350545: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fc8a4771c00 of size 4194304 | |
| 2018-12-03 00:43:03.350565: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a4b71c00 of size 4194304 | |
| 2018-12-03 00:43:03.350585: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a4f71c00 of size 4194304 | |
| 2018-12-03 00:43:03.350605: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a5371c00 of size 4194304 | |
| 2018-12-03 00:43:03.350625: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fc8a5771c00 of size 56803328 | |
| 2018-12-03 00:43:03.350646: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a8d9dc00 of size 56803328 | |
| 2018-12-03 00:43:03.350666: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fc8ac3c9c00 of size 56803328 | |
| 2018-12-03 00:43:03.350685: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8af9f5c00 of size 56803328 | |
| 2018-12-03 00:43:03.350704: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8b3021c00 of size 3077142784 | |
| 2018-12-03 00:43:03.350724: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc96a6b9500 of size 3077142784 | |
| 2018-12-03 00:43:03.350745: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fca21d50e00 of size 1613780992 | |
| 2018-12-03 00:43:03.350765: I tensorflow/core/common_runtime/bfc_allocator.cc:671] Summary of in-use Chunks by size: | |
| 2018-12-03 00:43:03.350795: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 76 Chunks of size 256 totalling 19.0KiB | |
| 2018-12-03 00:43:03.350800: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1024 totalling 1.0KiB | |
| 2018-12-03 00:43:03.350805: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 1280 totalling 2.5KiB | |
| 2018-12-03 00:43:03.350811: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 32 Chunks of size 2048 totalling 64.0KiB | |
| 2018-12-03 00:43:03.350815: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 4096 totalling 4.0KiB | |
| 2018-12-03 00:43:03.350820: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 3 Chunks of size 8192 totalling 24.0KiB | |
| 2018-12-03 00:43:03.350826: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 10240 totalling 10.0KiB | |
| 2018-12-03 00:43:03.350831: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 12032 totalling 11.8KiB | |
| 2018-12-03 00:43:03.350836: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 12288 totalling 12.0KiB | |
| 2018-12-03 00:43:03.350841: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 14 Chunks of size 32768 totalling 448.0KiB | |
| 2018-12-03 00:43:03.350847: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 34816 totalling 68.0KiB | |
| 2018-12-03 00:43:03.350852: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 65536 totalling 64.0KiB | |
| 2018-12-03 00:43:03.350857: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 71680 totalling 70.0KiB | |
| 2018-12-03 00:43:03.350862: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 111104 totalling 108.5KiB | |
| 2018-12-03 00:43:03.350867: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 17 Chunks of size 1048576 totalling 17.00MiB | |
| 2018-12-03 00:43:03.350873: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1050624 totalling 1.00MiB | |
| 2018-12-03 00:43:03.350878: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 1081344 totalling 2.06MiB | |
| 2018-12-03 00:43:03.350883: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1955840 totalling 1.87MiB | |
| 2018-12-03 00:43:03.350892: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1966080 totalling 1.88MiB | |
| 2018-12-03 00:43:03.350897: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1990656 totalling 1.90MiB | |
| 2018-12-03 00:43:03.350903: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 2056192 totalling 1.96MiB | |
| 2018-12-03 00:43:03.350908: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 11 Chunks of size 4194304 totalling 44.00MiB | |
| 2018-12-03 00:43:03.350912: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 4202496 totalling 4.01MiB | |
| 2018-12-03 00:43:03.350918: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 56803328 totalling 108.34MiB | |
| 2018-12-03 00:43:03.350923: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 3077142784 totalling 5.73GiB | |
| 2018-12-03 00:43:03.350928: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Sum Total of in-use chunks: 5.91GiB | |
| 2018-12-03 00:43:03.350935: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats: | |
| Limit: 8086972006 | |
| InUse: 6348168448 | |
| MaxInUse: 7654274048 | |
| NumAllocs: 5697987 | |
| MaxAllocSize: 3077142784 | |
| 2018-12-03 00:43:03.350949: W tensorflow/core/common_runtime/bfc_allocator.cc:279] *********************************************************************************___________________ | |
| 2018-12-03 00:43:03.350977: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at softmax_op_gpu.cu.cc:157 : Resource exhausted: OOM when allocating tensor with shape[221888,3467] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc | |
| Traceback (most recent call last): | |
| File "/usr/local/bin/t2t-trainer", line 32, in <module> | |
| tf.app.run() | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 126, in run | |
| _sys.exit(main(argv)) | |
| File "/usr/local/bin/t2t-trainer", line 28, in main | |
| t2t_trainer.main(argv) | |
| File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/bin/t2t_trainer.py", line 359, in main | |
| execute_schedule(exp) | |
| File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/bin/t2t_trainer.py", line 306, in execute_schedule | |
| getattr(exp, FLAGS.schedule)() | |
| File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/trainer_lib.py", line 289, in continuous_train_and_eval | |
| self._eval_spec) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 439, in train_and_evaluate | |
| executor.run() | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 518, in run | |
| self.run_local() | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 657, in run_local | |
| eval_result = evaluator.evaluate_and_export() | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 847, in evaluate_and_export | |
| hooks=self._eval_spec.hooks) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 425, in evaluate | |
| name=name) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 1117, in _evaluate_model | |
| config=self._session_config) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/evaluation.py", line 212, in _evaluate_once | |
| session.run(eval_ops, feed_dict) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 567, in run | |
| run_metadata=run_metadata) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1043, in run | |
| run_metadata=run_metadata) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1134, in run | |
| raise six.reraise(*original_exc_info) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1119, in run | |
| return self._sess.run(*args, **kwargs) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1191, in run | |
| run_metadata=run_metadata) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 971, in run | |
| return self._sess.run(*args, **kwargs) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 900, in run | |
| run_metadata_ptr) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1135, in _run | |
| feed_dict_tensor, options, run_metadata) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1316, in _do_run | |
| run_metadata) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1335, in _do_call | |
| raise type(e)(node_def, op, message) | |
| tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[221888,3467] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc | |
| [[Node: transformer_encoder/parallel_0_5/transformer_encoder/transformer_encoder/body/encoder/layer_0/self_attention/multihead_attention/dot_product_attention/Softmax = Softmax[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](transformer_encoder/parallel_0_5/transformer_encoder/transformer_encoder/body/encoder/layer_0/self_attention/multihead_attention/dot_product_attention/Reshape)]] | |
| Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. | |
| [[Node: transformer_encoder/parallel_0_5/transformer_encoder/transformer_encoder/body/encoder/layer_3/self_attention/multihead_attention/q/Tensordot/Shape/_775 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1161_...rdot/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]] | |
| Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. | |
| Caused by op u'transformer_encoder/parallel_0_5/transformer_encoder/transformer_encoder/body/encoder/layer_0/self_attention/multihead_attention/dot_product_attention/Softmax', defined at: | |
| File "/usr/local/bin/t2t-trainer", line 32, in <module> | |
| tf.app.run() | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 126, in run | |
| _sys.exit(main(argv)) | |
| File "/usr/local/bin/t2t-trainer", line 28, in main | |
| t2t_trainer.main(argv) | |
| File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/bin/t2t_trainer.py", line 359, in main | |
| execute_schedule(exp) | |
| File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/bin/t2t_trainer.py", line 306, in execute_schedule | |
| getattr(exp, FLAGS.schedule)() | |
| File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/trainer_lib.py", line 289, in continuous_train_and_eval | |
| self._eval_spec) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 439, in train_and_evaluate | |
| executor.run() | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 518, in run | |
| self.run_local() | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 657, in run_local | |
| eval_result = evaluator.evaluate_and_export() | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 847, in evaluate_and_export | |
| hooks=self._eval_spec.hooks) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 425, in evaluate | |
| name=name) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 1087, in _evaluate_model | |
| features, labels, model_fn_lib.ModeKeys.EVAL, self.config) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 831, in _call_model_fn | |
| model_fn_results = self._model_fn(features=features, **kwargs) | |
| File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 1155, in wrapping_model_fn | |
| use_tpu=use_tpu) | |
| File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 1206, in estimator_model_fn | |
| logits, losses_dict = model(features) # pylint: disable=not-callable | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/base.py", line 717, in __call__ | |
| outputs = self.call(inputs, *args, **kwargs) | |
| File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 176, in call | |
| sharded_logits, losses = self.model_fn_sharded(sharded_features) | |
| File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 231, in model_fn_sharded | |
| sharded_logits, sharded_losses = dp(self.model_fn, datashard_to_features) | |
| File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/expert_utils.py", line 231, in __call__ | |
| outputs.append(fns[i](*my_args[i], **my_kwargs[i])) | |
| File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 265, in model_fn | |
| body_out = self.body(transformed_features) | |
| File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py", line 1017, in body | |
| nonpadding=features_to_nonpadding(features, "inputs")) | |
| File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py", line 1220, in transformer_encoder | |
| vars_3d=hparams.get("attention_variables_3d")) | |
| File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/common_attention.py", line 2944, in multihead_attention | |
| dropout_broadcast_dims=dropout_broadcast_dims) | |
| File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/common_attention.py", line 1490, in dot_product_attention | |
| weights = tf.nn.softmax(logits, name="attention_weights") | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 432, in new_func | |
| return func(*args, **kwargs) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 1738, in softmax | |
| return _softmax(logits, gen_nn_ops.softmax, axis, name) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 1680, in _softmax | |
| output = compute_op(logits) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 7097, in softmax | |
| "Softmax", logits=logits, name=name) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper | |
| op_def=op_def) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3392, in create_op | |
| op_def=op_def) | |
| File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1718, in __init__ | |
| self._traceback = self._graph._extract_stack() # pylint: disable=protected-access | |
| ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[221888,3467] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc | |
| [[Node: transformer_encoder/parallel_0_5/transformer_encoder/transformer_encoder/body/encoder/layer_0/self_attention/multihead_attention/dot_product_attention/Softmax = Softmax[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](transformer_encoder/parallel_0_5/transformer_encoder/transformer_encoder/body/encoder/layer_0/self_attention/multihead_attention/dot_product_attention/Reshape)]] | |
| Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. | |
| [[Node: transformer_encoder/parallel_0_5/transformer_encoder/transformer_encoder/body/encoder/layer_3/self_attention/multihead_attention/q/Tensordot/Shape/_775 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1161_...rdot/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]] | |
| Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment