Last active
December 30, 2019 07:46
-
-
Save manish-kumar-garg/7b017cfed5827a9083d5e16f7df43fda to your computer and use it in GitHub Desktop.
asr local attention error
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| warning: unable to access '/home/ubuntu/.gitconfig': Is a directory | |
| warning: unable to access '/home/ubuntu/.gitconfig': Is a directory | |
| fatal: unknown error occurred while reading the configuration files | |
| RETURNN starting up, version unknown(git exception: CalledProcessError(128, ('git', 'show', '-s', '--format=%ci', 'HEAD'))), date/time 2019-12-30-07-45-03 (UTC+0000), pid 61826, cwd /home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention, Python /home/ubuntu/tf1.13/bin/python3 | |
| RETURNN command line options: ['local_win05.config'] | |
| Hostname: ip-10-1-21-241 | |
| /home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. | |
| _np_qint8 = np.dtype([("qint8", np.int8, 1)]) | |
| /home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. | |
| _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) | |
| /home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. | |
| _np_qint16 = np.dtype([("qint16", np.int16, 1)]) | |
| /home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. | |
| _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) | |
| /home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. | |
| _np_qint32 = np.dtype([("qint32", np.int32, 1)]) | |
| /home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. | |
| np_resource = np.dtype([("resource", np.ubyte, 1)]) | |
| TensorFlow: 1.13.1 (b'v1.13.1-0-g6612da8951') (<site-package> in /home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow) | |
| Setup TF inter and intra global thread pools, num_threads None, session opts {'log_device_placement': False, 'device_count': {'GPU': 0}}. | |
| 2019-12-30 07:45:04.884803: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA | |
| 2019-12-30 07:45:05.081329: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero | |
| 2019-12-30 07:45:05.083163: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55ef8d5e1630 executing computations on platform CUDA. Devices: | |
| 2019-12-30 07:45:05.083215: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla K80, Compute Capability 3.7 | |
| 2019-12-30 07:45:05.106530: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300090000 Hz | |
| 2019-12-30 07:45:05.109077: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55ef8dca4bd0 executing computations on platform Host. Devices: | |
| 2019-12-30 07:45:05.109120: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined> | |
| 2019-12-30 07:45:05.109246: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: | |
| 2019-12-30 07:45:05.109277: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] | |
| CUDA_VISIBLE_DEVICES is set to '1'. | |
| Collecting TensorFlow device list... | |
| 2019-12-30 07:45:05.112615: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: | |
| name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235 | |
| pciBusID: 0000:00:18.0 | |
| totalMemory: 11.17GiB freeMemory: 11.11GiB | |
| 2019-12-30 07:45:05.112650: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0 | |
| 2019-12-30 07:45:05.114885: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: | |
| 2019-12-30 07:45:05.114911: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 | |
| 2019-12-30 07:45:05.114928: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N | |
| 2019-12-30 07:45:05.115083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:0 with 10805 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:18.0, compute capability: 3.7) | |
| Local devices available to TensorFlow: | |
| 1/4: name: "/device:CPU:0" | |
| device_type: "CPU" | |
| memory_limit: 268435456 | |
| locality { | |
| } | |
| incarnation: 8403144321325319463 | |
| 2/4: name: "/device:XLA_GPU:0" | |
| device_type: "XLA_GPU" | |
| memory_limit: 17179869184 | |
| locality { | |
| } | |
| incarnation: 10706382646999000709 | |
| physical_device_desc: "device: XLA_GPU device" | |
| 3/4: name: "/device:XLA_CPU:0" | |
| device_type: "XLA_CPU" | |
| memory_limit: 17179869184 | |
| locality { | |
| } | |
| incarnation: 1490385100323021608 | |
| physical_device_desc: "device: XLA_CPU device" | |
| 4/4: name: "/device:GPU:0" | |
| device_type: "GPU" | |
| memory_limit: 11330115994 | |
| locality { | |
| bus_id: 1 | |
| links { | |
| } | |
| } | |
| incarnation: 9325274229127908753 | |
| physical_device_desc: "device: 0, name: Tesla K80, pci bus id: 0000:00:18.0, compute capability: 3.7" | |
| Using gpu device 1: Tesla K80 | |
| use local file: data/dataset/dev-clean.zip | |
| use local file: data/dataset/dev-other.zip | |
| use local file: data/dataset/train-clean-100.zip | |
| use local file: data/dataset/train-clean-360.zip | |
| use local file: data/dataset/train-other-500.zip | |
| <LibriSpeechCorpus 'train' epoch=1>, epoch 1. Old mean seq len (transcription) is 183.267376, new is 63.708029, requested max is 75.000000. Old num seqs is 6575, new num seqs is 822. | |
| <LibriSpeechCorpus 'train' epoch=1>, epoch 1. Old num seqs 14063, new num seqs 822. | |
| <LibriSpeechCorpus 'train' epoch=1>, epoch 1. Old mean seq len (transcription) is 183.267376, new is 63.708029, requested max is 75.000000. Old num seqs is 6575, new num seqs is 822. | |
| <LibriSpeechCorpus 'train' epoch=1>, epoch 1. Old num seqs 14063, new num seqs 822. | |
| Train data: | |
| input: 40 x 1 | |
| output: {'classes': [10025, 1], 'raw': {'dtype': 'string', 'shape': ()}, 'data': [40, 2]} | |
| LibriSpeechCorpus, sequences: 822, frames: unknown | |
| Dev data: | |
| LibriSpeechCorpus, sequences: 3000, frames: unknown | |
| Learning-rate-control: file data/exp-local_win05/train-scores.data does not exist yet | |
| Update config key 'max_seq_length' for epoch 1: {'classes': 75} -> {'classes': 60} | |
| Setup tf.Session with options {'log_device_placement': False, 'device_count': {'GPU': 1}} ... | |
| 2019-12-30 07:45:11.119403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0 | |
| 2019-12-30 07:45:11.119463: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: | |
| 2019-12-30 07:45:11.119479: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 | |
| 2019-12-30 07:45:11.119493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N | |
| 2019-12-30 07:45:11.119659: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10805 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:18.0, compute capability: 3.7) | |
| layer root/'data' output: Data(name='data', shape=(None, 40), batch_shape_meta=[B,T|'time:var:extern_data:data',F|40]) | |
| layer root/'source' output: Data(name='source_output', shape=(None, 40), batch_shape_meta=[B,T|'time:var:extern_data:data',F|40]) | |
| layer root/'lstm0_fw' output: Data(name='lstm0_fw_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'time:var:extern_data:data',B,F|1024]) | |
| layer root/'lstm0_bw' output: Data(name='lstm0_bw_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'time:var:extern_data:data',B,F|1024]) | |
| layer root/'lstm0_pool' output: Data(name='lstm0_pool_output', shape=(None, 2048), batch_shape_meta=[B,T|?,F|2048]) | |
| layer root/'lstm5_fw' output: Data(name='lstm5_fw_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm0_pool',B,F|1024]) | |
| layer root/'lstm5_bw' output: Data(name='lstm5_bw_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm0_pool',B,F|1024]) | |
| layer root/'encoder' output: Data(name='encoder_output', shape=(None, 2048), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm0_pool',B,F|2048]) | |
| layer root/'ctc' output: Data(name='ctc_output', shape=(None, 10026), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm0_pool',B,F|10026]) | |
| layer root/'enc_ctx' output: Data(name='enc_ctx_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm0_pool',B,F|1024]) | |
| layer root/'inv_fertility' output: Data(name='inv_fertility_output', shape=(None, 1), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm0_pool',B,F|1]) | |
| layer root/'enc_value' output: Data(name='enc_value_output', shape=(None, 1, 2048), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm0_pool',B,1,F|2048]) | |
| layer root/'output' output: Data(name='output_output', shape=(None,), dtype='int32', sparse=True, dim=10025, batch_dim_axis=1, batch_shape_meta=[T|'time:var:extern_data:classes',B]) | |
| Rec layer 'output' (search False, train 'globals/train_flag:0') sub net: | |
| Input layers moved out of loop: (#: 2) | |
| output | |
| target_embed | |
| Output layers moved out of loop: (#: 3) | |
| output_prob | |
| readout | |
| readout_in | |
| Layers in loop: (#: 12) | |
| s | |
| att | |
| att0 | |
| att_weights | |
| energy_reinterpreted | |
| energy | |
| energy_tanh | |
| energy_in | |
| weight_feedback | |
| accum_att_weights | |
| s_transformed | |
| p_t_in | |
| Unused layers: (#: 2) | |
| end | |
| p_t | |
| layer root/output:rec-subnet-input/'output' output: Data(name='output_output', shape=(None,), dtype='int32', sparse=True, dim=10025, batch_shape_meta=[B,T|'time:var:extern_data:classes']) | |
| layer root/output:rec-subnet-input/'target_embed' output: Data(name='target_embed_output', shape=(None, 621), batch_shape_meta=[B,T|'time:var:extern_data:classes',F|621]) | |
| layer root/output:rec-subnet/'weight_feedback' output: Data(name='weight_feedback_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|?,B,F|1024]) | |
| layer root/output:rec-subnet/'prev:target_embed' output: Data(name='target_embed_output', shape=(621,), time_dim_axis=None, batch_shape_meta=[B,F|621]) | |
| layer root/output:rec-subnet/'s' output: Data(name='s_output', shape=(1000,), time_dim_axis=None, batch_shape_meta=[B,F|1000]) | |
| layer root/output:rec-subnet/'s_transformed' output: Data(name='s_transformed_output', shape=(1024,), time_dim_axis=None, batch_shape_meta=[B,F|1024]) | |
| layer root/output:rec-subnet/'energy_in' output: Data(name='energy_in_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm0_pool',B,F|1024]) | |
| layer root/output:rec-subnet/'energy_tanh' output: Data(name='energy_tanh_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm0_pool',B,F|1024]) | |
| layer root/output:rec-subnet/'energy' output: Data(name='energy_output', shape=(None, 1), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm0_pool',B,F|1]) | |
| layer root/output:rec-subnet/'energy_reinterpreted' output: Data(name='energy_reinterpreted_output', shape=(None, 1), batch_shape_meta=[B,T|'spatial:0:lstm0_pool',F|1]) | |
| layer root/output:rec-subnet/'p_t_in' output: Data(name='p_t_in_output', shape=(1,), dtype='int32', sparse=True, dim=None, time_dim_axis=None, feature_dim_axis=None, batch_shape_meta=[B,1]) | |
| layer root/output:rec-subnet/'att_weights' output: Data(name='att_weights_output', shape=(1, None), time_dim_axis=2, feature_dim_axis=1, batch_shape_meta=[B,F|1,T|'spatial:0:lstm0_pool']) | |
| layer root/output:rec-subnet/'att0' output: Data(name='att0_output', shape=(1, 2048), time_dim_axis=None, batch_shape_meta=[B,1,F|2048]) | |
| layer root/output:rec-subnet/'att' output: Data(name='att_output', shape=(2048,), time_dim_axis=None, batch_shape_meta=[B,F|2048]) | |
| layer root/output:rec-subnet/'accum_att_weights' output: Data(name='accum_att_weights_output', shape=(None, 1), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm0_pool',B,F|1]) | |
| layer root/output:rec-subnet-output/'s' output: Data(name='s_output', shape=(None, 1000), batch_dim_axis=1, batch_shape_meta=[T|'time:var:extern_data:classes',B,F|1000]) | |
| layer root/output:rec-subnet-output/'prev:target_embed' output: Data(name='target_embed_output', shape=(None, 621), batch_dim_axis=1, batch_shape_meta=[T|'time:var:extern_data:classes',B,F|621]) | |
| layer root/output:rec-subnet-output/'att' output: Data(name='att_output', shape=(None, 2048), batch_dim_axis=1, batch_shape_meta=[T|'time:var:extern_data:classes',B,F|2048]) | |
| layer root/output:rec-subnet-output/'readout_in' output: Data(name='readout_in_output', shape=(None, 1000), batch_dim_axis=1, batch_shape_meta=[T|'time:var:extern_data:classes',B,F|1000]) | |
| layer root/output:rec-subnet-output/'readout' output: Data(name='readout_output', shape=(None, 500), batch_dim_axis=1, batch_shape_meta=[T|'time:var:extern_data:classes',B,F|500]) | |
| layer root/output:rec-subnet-output/'output_prob' output: Data(name='output_prob_output', shape=(None, 10025), batch_dim_axis=1, batch_shape_meta=[T|'time:var:extern_data:classes',B,F|10025]) | |
| layer root/'decision' output: Data(name='output_output', shape=(None,), dtype='int32', sparse=True, dim=10025, batch_dim_axis=1, batch_shape_meta=[T|'time:var:extern_data:classes',B]) | |
| Network layer topology: | |
| extern data: classes: Data(shape=(None,), dtype='int32', sparse=True, dim=10025, available_for_inference=False, batch_shape_meta=[B,T|'time:var:extern_data:classes']), data: Data(shape=(None, 40), batch_shape_meta=[B,T|'time:var:extern_data:data',F|40]) | |
| used data keys: ['classes', 'data'] | |
| layers: | |
| layer softmax 'ctc' #: 10026 | |
| layer source 'data' #: 40 | |
| layer decide 'decision' #: 10025 | |
| layer linear 'enc_ctx' #: 1024 | |
| layer split_dims 'enc_value' #: 2048 | |
| layer copy 'encoder' #: 2048 | |
| layer linear 'inv_fertility' #: 1 | |
| layer rec 'lstm0_bw' #: 1024 | |
| layer rec 'lstm0_fw' #: 1024 | |
| layer pool 'lstm0_pool' #: 2048 | |
| layer rec 'lstm5_bw' #: 1024 | |
| layer rec 'lstm5_fw' #: 1024 | |
| layer rec 'output' #: 10025 | |
| layer eval 'source' #: 40 | |
| net params #: 87166092 | |
| net trainable params: [<tf.Variable 'ctc/W:0' shape=(2048, 10026) dtype=float32_ref>, <tf.Variable 'ctc/b:0' shape=(10026,) dtype=float32_ref>, <tf.Variable 'enc_ctx/W:0' shape=(2048, 1024) dtype=float32_ref>, <tf.Variable 'enc_ctx/b:0' shape=(1024,) dtype=float32_ref>, <tf.Variable 'inv_fertility/W:0' shape=(2048, 1) dtype=float32_ref>, <tf.Variable 'lstm0_bw/rec/W:0' shape=(40, 4096) dtype=float32_ref>, <tf.Variable 'lstm0_bw/rec/W_re:0' shape=(1024, 4096) dtype=float32_ref>, <tf.Variable 'lstm0_bw/rec/b:0' shape=(4096,) dtype=float32_ref>, <tf.Variable 'lstm0_fw/rec/W:0' shape=(40, 4096) dtype=float32_ref>, <tf.Variable 'lstm0_fw/rec/W_re:0' shape=(1024, 4096) dtype=float32_ref>, <tf.Variable 'lstm0_fw/rec/b:0' shape=(4096,) dtype=float32_ref>, <tf.Variable 'lstm5_bw/rec/W:0' shape=(2048, 4096) dtype=float32_ref>, <tf.Variable 'lstm5_bw/rec/W_re:0' shape=(1024, 4096) dtype=float32_ref>, <tf.Variable 'lstm5_bw/rec/b:0' shape=(4096,) dtype=float32_ref>, <tf.Variable 'lstm5_fw/rec/W:0' shape=(2048, 4096) dtype=float32_ref>, <tf.Variable 'lstm5_fw/rec/W_re:0' shape=(1024, 4096) dtype=float32_ref>, <tf.Variable 'lstm5_fw/rec/b:0' shape=(4096,) dtype=float32_ref>, <tf.Variable 'output/rec/energy/W:0' shape=(1024, 1) dtype=float32_ref>, <tf.Variable 'output/rec/output_prob/W:0' shape=(500, 10025) dtype=float32_ref>, <tf.Variable 'output/rec/output_prob/b:0' shape=(10025,) dtype=float32_ref>, <tf.Variable 'output/rec/readout_in/W:0' shape=(3669, 1000) dtype=float32_ref>, <tf.Variable 'output/rec/readout_in/b:0' shape=(1000,) dtype=float32_ref>, <tf.Variable 'output/rec/s/rec/lstm_cell/bias:0' shape=(4000,) dtype=float32_ref>, <tf.Variable 'output/rec/s/rec/lstm_cell/kernel:0' shape=(3669, 4000) dtype=float32_ref>, <tf.Variable 'output/rec/s_transformed/W:0' shape=(1000, 1024) dtype=float32_ref>, <tf.Variable 'output/rec/target_embed/W:0' shape=(10025, 621) dtype=float32_ref>, <tf.Variable 'output/rec/weight_feedback/W:0' shape=(1, 1024) dtype=float32_ref>] | |
| start training at epoch 1 and step 0 | |
| using batch size: 20000, max seqs: 200 | |
| learning rate control: NewbobMultiEpoch(num_epochs=2, update_interval=1, relative_error_threshold=-0.01, learning_rate_decay_factor=0.9, learning_rate_growth_factor=1.0), epoch data: , error key: None | |
| pretrain: Pretrain construction algo <function custom_construction_algo at 0x7fb3921b4ae8>, number of pretrain epochs: 45 (repetitions: [10, 5, 5, 5, 5, 5, 5, 5]) | |
| <LibriSpeechCorpus 'train' epoch=1>, epoch 1. Old mean seq len (transcription) is 183.267376, new is 63.708029, requested max is 75.000000. Old num seqs is 6575, new num seqs is 822. | |
| <LibriSpeechCorpus 'train' epoch=1>, epoch 1. Old num seqs 14063, new num seqs 822. | |
| Update config key 'max_seq_length' for epoch 1: {'classes': 75} -> {'classes': 60} | |
| start pretrain epoch 1 with learning rate 1e-05 ... | |
| TF: log_dir: data/exp-local_win05/train-2019-12-30-07-45-03 | |
| Create Adam optimizer. | |
| /home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:110: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. | |
| "Converting sparse IndexedSlices to a dense Tensor of unknown shape. " | |
| Initialize optimizer (default) with slots ['m', 'v']. | |
| These additional variable were created by the optimizer: [<tf.Variable 'optimize/beta1_power:0' shape=() dtype=float32_ref>, <tf.Variable 'optimize/beta2_power:0' shape=() dtype=float32_ref>]. | |
| 2019-12-30 07:45:27.482527: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally | |
| TensorFlow exception: Incompatible shapes: [14,1,45] vs. [45,1,45] | |
| [[node output/rec/att_weights/LogicalAnd_1 (defined at /home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkLayer.py:3159) ]] | |
| Caused by op 'output/rec/att_weights/LogicalAnd_1', defined at: | |
| File "./returnn/rnn.py", line 654, in <module> | |
| main(sys.argv) | |
| File "./returnn/rnn.py", line 642, in main | |
| execute_main_task() | |
| File "./returnn/rnn.py", line 451, in execute_main_task | |
| engine.init_train_from_config(config, train_data, dev_data, eval_data) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFEngine.py", line 891, in init_train_from_config | |
| self.init_network_from_config(config) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFEngine.py", line 934, in init_network_from_config | |
| self._init_network(net_desc=net_dict, epoch=self.epoch) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFEngine.py", line 1081, in _init_network | |
| net_dict=net_desc) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFEngine.py", line 1113, in create_network | |
| network.construct_from_dict(net_dict) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 460, in construct_from_dict | |
| self.construct_layer(net_dict, name) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 652, in construct_layer | |
| layer_class.transform_config_dict(layer_desc, network=self, get_layer=get_layer) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 4092, in transform_config_dict | |
| super(BaseChoiceLayer, cls).transform_config_dict(d, network=network, get_layer=get_layer) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkLayer.py", line 448, in transform_config_dict | |
| for src_name in src_names | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkLayer.py", line 449, in <listcomp> | |
| if not src_name == "none"] | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 607, in get_layer | |
| return self.construct_layer(net_dict=net_dict, name=src_name) # set get_layer to wrap construct_layer | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 655, in construct_layer | |
| return add_layer(name=name, layer_class=layer_class, **layer_desc) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 760, in add_layer | |
| layer = self._create_layer(name=name, layer_class=layer_class, **layer_desc) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 709, in _create_layer | |
| layer = layer_class(**layer_desc) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 210, in __init__ | |
| y = self._get_output_subnet_unit(self.cell) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 834, in _get_output_subnet_unit | |
| output = cell.get_output(rec_layer=self) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 2252, in get_output | |
| shape_invariants=shape_invariants) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 1627, in _while_loop | |
| back_prop=self.parent_rec_layer.back_prop) | |
| File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3556, in while_loop | |
| return_same_structure) | |
| File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3087, in BuildLoop | |
| pred, body, original_loop_vars, loop_vars, shape_invariants) | |
| File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3022, in _BuildLoop | |
| body_result = body(*packed_vars_for_body) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 2102, in body | |
| needed_outputs=needed_outputs) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 1422, in _construct | |
| get_layer(layer_name) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 1398, in get_layer | |
| layer = self.net.construct_layer(net_dict, name=name, get_layer=get_layer) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 652, in construct_layer | |
| layer_class.transform_config_dict(layer_desc, network=self, get_layer=get_layer) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkLayer.py", line 448, in transform_config_dict | |
| for src_name in src_names | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkLayer.py", line 449, in <listcomp> | |
| if not src_name == "none"] | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 1398, in get_layer | |
| layer = self.net.construct_layer(net_dict, name=name, get_layer=get_layer) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 652, in construct_layer | |
| layer_class.transform_config_dict(layer_desc, network=self, get_layer=get_layer) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 5224, in transform_config_dict | |
| d["weights"] = get_layer(d["weights"]) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 1398, in get_layer | |
| layer = self.net.construct_layer(net_dict, name=name, get_layer=get_layer) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 655, in construct_layer | |
| return add_layer(name=name, layer_class=layer_class, **layer_desc) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 760, in add_layer | |
| layer = self._create_layer(name=name, layer_class=layer_class, **layer_desc) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 709, in _create_layer | |
| layer = layer_class(**layer_desc) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkLayer.py", line 3003, in __init__ | |
| window_size=window_size.output if isinstance(window_size, LayerBase) else window_size) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkLayer.py", line 3159, in build_mask | |
| tf.less(idxs, window_start.placeholder + window_size.placeholder) | |
| File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 5095, in logical_and | |
| "LogicalAnd", x=x, y=y, name=name) | |
| File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper | |
| op_def=op_def) | |
| File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func | |
| return func(*args, **kwargs) | |
| File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op | |
| op_def=op_def) | |
| File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__ | |
| self._traceback = tf_stack.extract_stack() | |
| InvalidArgumentError (see above for traceback): Incompatible shapes: [14,1,45] vs. [45,1,45] | |
| [[node output/rec/att_weights/LogicalAnd_1 (defined at /home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkLayer.py:3159) ]] | |
| Exception InvalidArgumentError() in step 0. | |
| Failing op: <tf.Operation 'output/rec/att_weights/LogicalAnd_1' type=LogicalAnd> | |
| Execute again to debug the op inputs... | |
| FetchHelper(0): <tf.Tensor 'output/rec/att_weights/LogicalAnd_2:0' shape=<unknown> dtype=bool> = shape (45, 1, 45), dtype bool, min/max False/True | |
| FetchHelper(0): <tf.Tensor 'output/rec/att_weights/Reshape_2:0' shape=(?, 1, ?) dtype=bool> = shape (14, 1, 45), dtype bool, min/max False/True | |
| Op inputs: | |
| <tf.Tensor 'output/rec/att_weights/Reshape:0' shape=(?, 1, ?) dtype=bool>: shape (14, 1, 45), dtype bool, min/max False/True | |
| <tf.Tensor 'output/rec/att_weights/LogicalAnd:0' shape=(?, 1, ?) dtype=bool>: shape (45, 1, 45), dtype bool, min/max False/True | |
| Step meta information: | |
| {'seq_idx': [0, | |
| 1, | |
| 2, | |
| 3, | |
| 4, | |
| 5, | |
| 6, | |
| 7, | |
| 8, | |
| 9, | |
| 10, | |
| 11, | |
| 12, | |
| 13, | |
| 14, | |
| 15, | |
| 16, | |
| 17, | |
| 18, | |
| 19, | |
| 20, | |
| 21, | |
| 22, | |
| 23, | |
| 24, | |
| 25, | |
| 26, | |
| 27, | |
| 28, | |
| 29, | |
| 30, | |
| 31, | |
| 32, | |
| 33, | |
| 34, | |
| 35, | |
| 36, | |
| 37, | |
| 38, | |
| 39, | |
| 40, | |
| 41, | |
| 42, | |
| 43, | |
| 44], | |
| 'seq_tag': ['train-clean-360-708-129393-0016', | |
| 'train-clean-360-4854-24732-0043', | |
| 'train-clean-360-3927-6557-0007', | |
| 'train-clean-360-4356-6506-0037', | |
| 'train-clean-360-3927-6556-0063', | |
| 'train-clean-100-3374-298026-0040', | |
| 'train-clean-360-8050-110238-0074', | |
| 'train-clean-360-8195-117382-0112', | |
| 'train-clean-360-671-131030-0024', | |
| 'train-clean-360-5448-19209-0024', | |
| 'train-clean-360-210-129396-0099', | |
| 'train-clean-360-8006-112593-0021', | |
| 'train-clean-360-210-129396-0126', | |
| 'train-clean-360-369-125882-0016', | |
| 'train-clean-360-2570-157243-0047', | |
| 'train-clean-360-7229-80665-0002', | |
| 'train-clean-360-209-4733-0022', | |
| 'train-clean-360-6918-47541-0068', | |
| 'train-clean-100-27-124992-0017', | |
| 'train-clean-360-210-129396-0078', | |
| 'train-clean-360-4363-14936-0014', | |
| 'train-clean-360-2285-163381-0010', | |
| 'train-clean-360-216-122451-0041', | |
| 'train-clean-100-1737-146161-0007', | |
| 'train-clean-100-2764-36616-0011', | |
| 'train-clean-360-1801-138032-0041', | |
| 'train-clean-360-2149-7239-0009', | |
| 'train-clean-360-2638-10172-0117', | |
| 'train-clean-360-8008-271812-0034', | |
| 'train-clean-100-328-129766-0017', | |
| 'train-clean-100-3259-158083-0039', | |
| 'train-clean-360-434-132649-0031', | |
| 'train-clean-360-434-132650-0013', | |
| 'train-clean-360-1974-139741-0052', | |
| 'train-clean-360-2368-157066-0001', | |
| 'train-clean-360-5133-30591-0042', | |
| 'train-clean-360-6497-234067-0018', | |
| 'train-clean-360-8050-110238-0068', | |
| 'train-clean-100-887-123290-0030', | |
| 'train-clean-360-5767-48579-0008', | |
| 'train-clean-360-6956-76046-0037', | |
| 'train-clean-360-525-126965-0115', | |
| 'train-clean-360-1705-142318-0072', | |
| 'train-clean-360-1943-138033-0065', | |
| 'train-clean-360-3448-5417-0009']} | |
| Feed dict: | |
| <tf.Tensor 'extern_data/placeholders/classes/classes:0' shape=(?, ?) dtype=int32>: shape (45, 12), dtype int32, min/max 0/9031, Data(name='classes', shape=(None,), dtype='int32', sparse=True, dim=10025, available_for_inference=False, batch_shape_meta=[B,T|'time:var:extern_data:classes']) | |
| <tf.Tensor 'extern_data/placeholders/classes/classes_dim0_size:0' shape=(?,) dtype=int32>: shape (45,), dtype int32, min/max 2/12, ([ 2 2 2 3 3 3 2 3 4 5 4 4 6 5 5 6 5 8 5 7 6 9 6 5 | |
| 6 6 7 7 7 6 7 6 6 7 8 7 6 6 7 6 8 12 9 7 8]) | |
| <tf.Tensor 'extern_data/placeholders/data/data:0' shape=(?, ?, 40) dtype=float32>: shape (45, 437, 40), dtype float32, min/max -5.747757/8.373278, mean/stddev 0.046555836/0.69089395, Data(name='data', shape=(None, 40), batch_shape_meta=[B,T|'time:var:extern_data:data',F|40]) | |
| <tf.Tensor 'extern_data/placeholders/data/data_dim0_size:0' shape=(?,) dtype=int32>: shape (45,), dtype int32, min/max 122/437, ([144 126 122 166 137 153 182 271 136 249 214 177 358 184 195 274 303 330 | |
| 270 363 237 346 208 204 194 202 194 224 273 202 216 237 221 376 437 296 | |
| 218 251 225 208 233 265 237 228 351]) | |
| <tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>: bool(True) | |
| EXCEPTION | |
| Traceback (most recent call last): | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFEngine.py", line 585, in run | |
| line: fetches_results = sess.run( | |
| fetches_dict, feed_dict=feed_dict) # type: typing.Dict[str,typing.Union[numpy.ndarray,str]] | |
| locals: | |
| fetches_results = <not found> | |
| sess = <local> <tensorflow.python.client.session.Session object at 0x7fb39a7ff630> | |
| sess.run = <local> <bound method BaseSession.run of <tensorflow.python.client.session.Session object at 0x7fb39a7ff630>> | |
| fetches_dict = <local> {'size:classes:0': <tf.Tensor 'extern_data/placeholders/classes/classes_dim0_size:0' shape=(?,) dtype=int32>, 'size:data:0': <tf.Tensor 'extern_data/placeholders/data/data_dim0_size:0' shape=(?,) dtype=int32>, 'loss': <tf.Tensor 'objective/add:0' shape=() dtype=float32>, 'cost:ctc': <tf.Tensor 'o..., len = 14 | |
| feed_dict = <local> {<tf.Tensor 'extern_data/placeholders/classes/classes:0' shape=(?, ?) dtype=int32>: array([[2085, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, | |
| 0], | |
| [1203, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, | |
| 0], | |
| [6137, 0, 0, 0, ... | |
| File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run | |
| line: result = self._run(None, fetches, feed_dict, options_ptr, | |
| run_metadata_ptr) | |
| locals: | |
| result = <not found> | |
| self = <local> <tensorflow.python.client.session.Session object at 0x7fb39a7ff630> | |
| self._run = <local> <bound method BaseSession._run of <tensorflow.python.client.session.Session object at 0x7fb39a7ff630>> | |
| fetches = <local> {'size:classes:0': <tf.Tensor 'extern_data/placeholders/classes/classes_dim0_size:0' shape=(?,) dtype=int32>, 'size:data:0': <tf.Tensor 'extern_data/placeholders/data/data_dim0_size:0' shape=(?,) dtype=int32>, 'loss': <tf.Tensor 'objective/add:0' shape=() dtype=float32>, 'cost:ctc': <tf.Tensor 'o..., len = 14 | |
| feed_dict = <local> {<tf.Tensor 'extern_data/placeholders/classes/classes:0' shape=(?, ?) dtype=int32>: array([[2085, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, | |
| 0], | |
| [1203, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, | |
| 0], | |
| [6137, 0, 0, 0, ... | |
| options_ptr = <local> None | |
| run_metadata_ptr = <local> None | |
| File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run | |
| line: results = self._do_run(handle, final_targets, final_fetches, | |
| feed_dict_tensor, options, run_metadata) | |
| locals: | |
| results = <not found> | |
| self = <local> <tensorflow.python.client.session.Session object at 0x7fb39a7ff630> | |
| self._do_run = <local> <bound method BaseSession._do_run of <tensorflow.python.client.session.Session object at 0x7fb39a7ff630>> | |
| handle = <local> None | |
| final_targets = <local> [<tf.Operation 'optim_and_step_incr' type=NoOp>] | |
| final_fetches = <local> [<tf.Tensor 'objective/add:0' shape=() dtype=float32>, <tf.Tensor 'objective/loss/loss/loss_ctc/Sum:0' shape=() dtype=float32>, <tf.Tensor 'objective/loss/error/loss_ctc_error/Sum:0' shape=() dtype=float32>, <tf.Tensor 'objective/loss/loss_init/truediv:0' shape=() dtype=float32>, <tf.Tensor 'obje..., len = 11 | |
| feed_dict_tensor = <local> {<tf.Tensor 'extern_data/placeholders/classes/classes:0' shape=(?, ?) dtype=int32>: array([[2085, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, | |
| 0], | |
| [1203, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, | |
| 0], | |
| [6137, 0, 0, 0, ... | |
| options = <local> None | |
| run_metadata = <local> None | |
| File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run | |
| line: return self._do_call(_run_fn, feeds, fetches, targets, options, | |
| run_metadata) | |
| locals: | |
| self = <local> <tensorflow.python.client.session.Session object at 0x7fb39a7ff630> | |
| self._do_call = <local> <bound method BaseSession._do_call of <tensorflow.python.client.session.Session object at 0x7fb39a7ff630>> | |
| _run_fn = <local> <function BaseSession._do_run.<locals>._run_fn at 0x7fb08df05378> | |
| feeds = <local> {<tensorflow.python.pywrap_tensorflow_internal.TF_Output; proxy of <Swig Object of type 'TF_Output *' at 0x7fb0e8ad8b10> >: array([[2085, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, | |
| 0], | |
| [1203, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, | |
| ... | |
| fetches = <local> [<tensorflow.python.pywrap_tensorflow_internal.TF_Output; proxy of <Swig Object of type 'TF_Output *' at 0x7fb0d93b6e10> >, <tensorflow.python.pywrap_tensorflow_internal.TF_Output; proxy of <Swig Object of type 'TF_Output *' at 0x7fb0d9a3e270> >, <tensorflow.python.pywrap_tensorflow_internal.TF_O..., len = 11 | |
| targets = <local> [<Swig Object of type 'TF_Operation *' at 0x7fb08dc24ba0>] | |
| options = <local> None | |
| run_metadata = <local> None | |
| File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call | |
| line: raise type(e)(node_def, op, message) | |
| locals: | |
| type = <builtin> <class 'type'> | |
| e = <not found> | |
| node_def = <local> name: "output/rec/att_weights/LogicalAnd_1" | |
| op: "LogicalAnd" | |
| input: "output/rec/att_weights/Reshape" | |
| input: "output/rec/att_weights/LogicalAnd" | |
| op = <local> <tf.Operation 'output/rec/att_weights/LogicalAnd_1' type=LogicalAnd> | |
| message = <local> 'Incompatible shapes: [14,1,45] vs. [45,1,45]\n\t [[node output/rec/att_weights/LogicalAnd_1 (defined at /home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkLayer.py:3159) ]]', len = 229 | |
| InvalidArgumentError: Incompatible shapes: [14,1,45] vs. [45,1,45] | |
| [[node output/rec/att_weights/LogicalAnd_1 (defined at /home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkLayer.py:3159) ]] | |
| Caused by op 'output/rec/att_weights/LogicalAnd_1', defined at: | |
| File "./returnn/rnn.py", line 654, in <module> | |
| main(sys.argv) | |
| File "./returnn/rnn.py", line 642, in main | |
| execute_main_task() | |
| File "./returnn/rnn.py", line 451, in execute_main_task | |
| engine.init_train_from_config(config, train_data, dev_data, eval_data) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFEngine.py", line 891, in init_train_from_config | |
| self.init_network_from_config(config) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFEngine.py", line 934, in init_network_from_config | |
| self._init_network(net_desc=net_dict, epoch=self.epoch) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFEngine.py", line 1081, in _init_network | |
| net_dict=net_desc) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFEngine.py", line 1113, in create_network | |
| network.construct_from_dict(net_dict) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 460, in construct_from_dict | |
| self.construct_layer(net_dict, name) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 652, in construct_layer | |
| layer_class.transform_config_dict(layer_desc, network=self, get_layer=get_layer) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 4092, in transform_config_dict | |
| super(BaseChoiceLayer, cls).transform_config_dict(d, network=network, get_layer=get_layer) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkLayer.py", line 448, in transform_config_dict | |
| for src_name in src_names | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkLayer.py", line 449, in <listcomp> | |
| if not src_name == "none"] | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 607, in get_layer | |
| return self.construct_layer(net_dict=net_dict, name=src_name) # set get_layer to wrap construct_layer | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 655, in construct_layer | |
| return add_layer(name=name, layer_class=layer_class, **layer_desc) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 760, in add_layer | |
| layer = self._create_layer(name=name, layer_class=layer_class, **layer_desc) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 709, in _create_layer | |
| layer = layer_class(**layer_desc) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 210, in __init__ | |
| y = self._get_output_subnet_unit(self.cell) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 834, in _get_output_subnet_unit | |
| output = cell.get_output(rec_layer=self) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 2252, in get_output | |
| shape_invariants=shape_invariants) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 1627, in _while_loop | |
| back_prop=self.parent_rec_layer.back_prop) | |
| File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3556, in while_loop | |
| return_same_structure) | |
| File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3087, in BuildLoop | |
| pred, body, original_loop_vars, loop_vars, shape_invariants) | |
| File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3022, in _BuildLoop | |
| body_result = body(*packed_vars_for_body) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 2102, in body | |
| needed_outputs=needed_outputs) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 1422, in _construct | |
| get_layer(layer_name) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 1398, in get_layer | |
| layer = self.net.construct_layer(net_dict, name=name, get_layer=get_layer) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 652, in construct_layer | |
| layer_class.transform_config_dict(layer_desc, network=self, get_layer=get_layer) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkLayer.py", line 448, in transform_config_dict | |
| for src_name in src_names | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkLayer.py", line 449, in <listcomp> | |
| if not src_name == "none"] | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 1398, in get_layer | |
| layer = self.net.construct_layer(net_dict, name=name, get_layer=get_layer) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 652, in construct_layer | |
| layer_class.transform_config_dict(layer_desc, network=self, get_layer=get_layer) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 5224, in transform_config_dict | |
| d["weights"] = get_layer(d["weights"]) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkRecLayer.py", line 1398, in get_layer | |
| layer = self.net.construct_layer(net_dict, name=name, get_layer=get_layer) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 655, in construct_layer | |
| return add_layer(name=name, layer_class=layer_class, **layer_desc) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 760, in add_layer | |
| layer = self._create_layer(name=name, layer_class=layer_class, **layer_desc) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 709, in _create_layer | |
| layer = layer_class(**layer_desc) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkLayer.py", line 3003, in __init__ | |
| window_size=window_size.output if isinstance(window_size, LayerBase) else window_size) | |
| File "/home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkLayer.py", line 3159, in build_mask | |
| tf.less(idxs, window_start.placeholder + window_size.placeholder) | |
| File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 5095, in logical_and | |
| "LogicalAnd", x=x, y=y, name=name) | |
| File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper | |
| op_def=op_def) | |
| File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func | |
| return func(*args, **kwargs) | |
| File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op | |
| op_def=op_def) | |
| File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__ | |
| self._traceback = tf_stack.extract_stack() | |
| InvalidArgumentError (see above for traceback): Incompatible shapes: [14,1,45] vs. [45,1,45] | |
| [[node output/rec/att_weights/LogicalAnd_1 (defined at /home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkLayer.py:3159) ]] | |
| Save model under data/exp-local_win05/model.pretrain.001.crash_0 | |
| Trainer not finalized, quitting. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment