Created
February 5, 2020 10:41
-
-
Save manish-kumar-garg/61b1863b1f55d7062e0dfac0f301f0de to your computer and use it in GitHub Desktop.
load_basiclstm_error
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Local devices available to TensorFlow: | |
| 1/4: name: "/device:CPU:0" | |
| device_type: "CPU" | |
| memory_limit: 268435456 | |
| locality { | |
| } | |
| incarnation: 12898452358282468420 | |
| 2/4: name: "/device:XLA_GPU:0" | |
| device_type: "XLA_GPU" | |
| memory_limit: 17179869184 | |
| locality { | |
| } | |
| incarnation: 17289528360015477638 | |
| physical_device_desc: "device: XLA_GPU device" | |
| 3/4: name: "/device:XLA_CPU:0" | |
| device_type: "XLA_CPU" | |
| memory_limit: 17179869184 | |
| locality { | |
| } | |
| incarnation: 6626347110952450108 | |
| physical_device_desc: "device: XLA_CPU device" | |
| 4/4: name: "/device:GPU:0" | |
| device_type: "GPU" | |
| memory_limit: 15782644941 | |
| locality { | |
| bus_id: 1 | |
| links { | |
| } | |
| } | |
| incarnation: 7695198597191475006 | |
| physical_device_desc: "device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1b.0, compute capability: 7.0" | |
| Using gpu device 0: Tesla V100-SXM2-16GB | |
| <LibriSpeechCorpus 'train' epoch=1>, epoch 1. Old mean seq len (transcription) is 183.267376, new is 63.708029, requested max is 75.000000. Old num seqs is 6575, new num seqs is 822. | |
| <LibriSpeechCorpus 'train' epoch=1>, epoch 1. Old num seqs 14063, new num seqs 822. | |
| <LibriSpeechCorpus 'train' epoch=1>, epoch 1. Old mean seq len (transcription) is 183.267376, new is 63.708029, requested max is 75.000000. Old num seqs is 6575, new num seqs is 822. | |
| <LibriSpeechCorpus 'train' epoch=1>, epoch 1. Old num seqs 14063, new num seqs 822. | |
| Train data: | |
| input: 40 x 1 | |
| output: {'classes': [10025, 1], 'raw': {'dtype': 'string', 'shape': ()}, 'data': [40, 2]} | |
| LibriSpeechCorpus, sequences: 822, frames: unknown | |
| Dev data: | |
| LibriSpeechCorpus, sequences: 3000, frames: unknown | |
| Learning-rate-control: file data/exp-returnn_basiclstm/train-scores.data does not exist yet | |
| Setup tf.Session with options {'log_device_placement': False, 'device_count': {'GPU': 1}} ... | |
| 2020-02-05 10:40:11.208016: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0 | |
| 2020-02-05 10:40:11.208071: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: | |
| 2020-02-05 10:40:11.208087: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 | |
| 2020-02-05 10:40:11.208095: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N | |
| 2020-02-05 10:40:11.208182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15051 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1b.0, compute capability: 7.0) | |
| layer root/'data' output: Data(name='data', shape=(None, 40), batch_shape_meta=[B,T|'time:var:extern_data:data',F|40]) | |
| layer root/'source' output: Data(name='source_output', shape=(None, 40), batch_shape_meta=[B,T|'time:var:extern_data:data',F|40]) | |
| layer root/'lstm0_fw' output: Data(name='lstm0_fw_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'time:var:extern_data:data',B,F|1024]) | |
| layer root/'lstm0_bw' output: Data(name='lstm0_bw_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'time:var:extern_data:data',B,F|1024]) | |
| layer root/'lstm0_pool' output: Data(name='lstm0_pool_output', shape=(None, 2048), batch_shape_meta=[B,T|?,F|2048]) | |
| layer root/'lstm1_fw' output: Data(name='lstm1_fw_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm0_pool',B,F|1024]) | |
| layer root/'lstm1_bw' output: Data(name='lstm1_bw_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm0_pool',B,F|1024]) | |
| layer root/'lstm1_pool' output: Data(name='lstm1_pool_output', shape=(None, 2048), batch_shape_meta=[B,T|?,F|2048]) | |
| layer root/'lstm2_fw' output: Data(name='lstm2_fw_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm1_pool',B,F|1024]) | |
| layer root/'lstm2_bw' output: Data(name='lstm2_bw_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm1_pool',B,F|1024]) | |
| layer root/'lstm2_pool' output: Data(name='lstm2_pool_output', shape=(None, 2048), batch_shape_meta=[B,T|?,F|2048]) | |
| layer root/'lstm3_fw' output: Data(name='lstm3_fw_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm2_pool',B,F|1024]) | |
| layer root/'lstm3_bw' output: Data(name='lstm3_bw_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm2_pool',B,F|1024]) | |
| layer root/'lstm3_pool' output: Data(name='lstm3_pool_output', shape=(None, 2048), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm2_pool',B,F|2048]) | |
| layer root/'lstm4_fw' output: Data(name='lstm4_fw_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm2_pool',B,F|1024]) | |
| layer root/'lstm4_bw' output: Data(name='lstm4_bw_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm2_pool',B,F|1024]) | |
| layer root/'lstm4_pool' output: Data(name='lstm4_pool_output', shape=(None, 2048), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm2_pool',B,F|2048]) | |
| layer root/'lstm5_fw' output: Data(name='lstm5_fw_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm2_pool',B,F|1024]) | |
| layer root/'lstm5_bw' output: Data(name='lstm5_bw_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm2_pool',B,F|1024]) | |
| layer root/'encoder' output: Data(name='encoder_output', shape=(None, 2048), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm2_pool',B,F|2048]) | |
| layer root/'ctc' output: Data(name='ctc_output', shape=(None, 10026), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm2_pool',B,F|10026]) | |
| layer root/'enc_ctx' output: Data(name='enc_ctx_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm2_pool',B,F|1024]) | |
| layer root/'inv_fertility' output: Data(name='inv_fertility_output', shape=(None, 1), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm2_pool',B,F|1]) | |
| layer root/'enc_value' output: Data(name='enc_value_output', shape=(None, 1, 2048), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm2_pool',B,1,F|2048]) | |
| layer root/'output' output: Data(name='output_output', shape=(None,), dtype='int32', sparse=True, dim=10025, batch_dim_axis=1, batch_shape_meta=[T|'time:var:extern_data:classes',B]) | |
| Rec layer 'output' (search False, train 'globals/train_flag:0') sub net: | |
| Input layers moved out of loop: (#: 2) | |
| output | |
| target_embed | |
| Output layers moved out of loop: (#: 3) | |
| output_prob | |
| readout | |
| readout_in | |
| Layers in loop: (#: 10) | |
| s | |
| att | |
| att0 | |
| att_weights | |
| energy | |
| energy_tanh | |
| energy_in | |
| weight_feedback | |
| accum_att_weights | |
| s_transformed | |
| Unused layers: (#: 1) | |
| end | |
| layer root/output:rec-subnet-input/'output' output: Data(name='output_output', shape=(None,), dtype='int32', sparse=True, dim=10025, batch_shape_meta=[B,T|'time:var:extern_data:classes']) | |
| layer root/output:rec-subnet-input/'target_embed' output: Data(name='target_embed_output', shape=(None, 621), batch_shape_meta=[B,T|'time:var:extern_data:classes',F|621]) | |
| layer root/output:rec-subnet/'weight_feedback' output: Data(name='weight_feedback_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|?,B,F|1024]) | |
| layer root/output:rec-subnet/'prev:target_embed' output: Data(name='target_embed_output', shape=(621,), time_dim_axis=None, batch_shape_meta=[B,F|621]) | |
| layer root/output:rec-subnet/'s' output: Data(name='s_output', shape=(1000,), time_dim_axis=None, batch_shape_meta=[B,F|1000]) | |
| layer root/output:rec-subnet/'s_transformed' output: Data(name='s_transformed_output', shape=(1024,), time_dim_axis=None, batch_shape_meta=[B,F|1024]) | |
| layer root/output:rec-subnet/'energy_in' output: Data(name='energy_in_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm2_pool',B,F|1024]) | |
| layer root/output:rec-subnet/'energy_tanh' output: Data(name='energy_tanh_output', shape=(None, 1024), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm2_pool',B,F|1024]) | |
| layer root/output:rec-subnet/'energy' output: Data(name='energy_output', shape=(None, 1), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm2_pool',B,F|1]) | |
| layer root/output:rec-subnet/'att_weights' output: Data(name='att_weights_output', shape=(1, None), time_dim_axis=2, feature_dim_axis=1, batch_shape_meta=[B,F|1,T|'spatial:0:lstm2_pool']) | |
| layer root/output:rec-subnet/'att0' output: Data(name='att0_output', shape=(1, 2048), time_dim_axis=None, batch_shape_meta=[B,1,F|2048]) | |
| layer root/output:rec-subnet/'att' output: Data(name='att_output', shape=(2048,), time_dim_axis=None, batch_shape_meta=[B,F|2048]) | |
| layer root/output:rec-subnet/'accum_att_weights' output: Data(name='accum_att_weights_output', shape=(None, 1), batch_dim_axis=1, batch_shape_meta=[T|'spatial:0:lstm2_pool',B,F|1]) | |
| layer root/output:rec-subnet-output/'s' output: Data(name='s_output', shape=(None, 1000), batch_dim_axis=1, batch_shape_meta=[T|'time:var:extern_data:classes',B,F|1000]) | |
| layer root/output:rec-subnet-output/'prev:target_embed' output: Data(name='target_embed_output', shape=(None, 621), batch_dim_axis=1, batch_shape_meta=[T|'time:var:extern_data:classes',B,F|621]) | |
| layer root/output:rec-subnet-output/'att' output: Data(name='att_output', shape=(None, 2048), batch_dim_axis=1, batch_shape_meta=[T|'time:var:extern_data:classes',B,F|2048]) | |
| layer root/output:rec-subnet-output/'readout_in' output: Data(name='readout_in_output', shape=(None, 1000), batch_dim_axis=1, batch_shape_meta=[T|'time:var:extern_data:classes',B,F|1000]) | |
| layer root/output:rec-subnet-output/'readout' output: Data(name='readout_output', shape=(None, 500), batch_dim_axis=1, batch_shape_meta=[T|'time:var:extern_data:classes',B,F|500]) | |
| layer root/output:rec-subnet-output/'output_prob' output: Data(name='output_prob_output', shape=(None, 10025), batch_dim_axis=1, batch_shape_meta=[T|'time:var:extern_data:classes',B,F|10025]) | |
| layer root/'decision' output: Data(name='output_output', shape=(None,), dtype='int32', sparse=True, dim=10025, batch_dim_axis=1, batch_shape_meta=[T|'time:var:extern_data:classes',B]) | |
| Network layer topology: | |
| extern data: classes: Data(shape=(None,), dtype='int32', sparse=True, dim=10025, available_for_inference=False, batch_shape_meta=[B,T|'time:var:extern_data:classes']), data: Data(shape=(None, 40), batch_shape_meta=[B,T|'time:var:extern_data:data',F|40]) | |
| used data keys: ['classes', 'data'] | |
| layers: | |
| layer softmax 'ctc' #: 10026 | |
| layer source 'data' #: 40 | |
| layer decide 'decision' #: 10025 | |
| layer linear 'enc_ctx' #: 1024 | |
| layer split_dims 'enc_value' #: 2048 | |
| layer copy 'encoder' #: 2048 | |
| layer linear 'inv_fertility' #: 1 | |
| layer rec 'lstm0_bw' #: 1024 | |
| layer rec 'lstm0_fw' #: 1024 | |
| layer pool 'lstm0_pool' #: 2048 | |
| layer rec 'lstm1_bw' #: 1024 | |
| layer rec 'lstm1_fw' #: 1024 | |
| layer pool 'lstm1_pool' #: 2048 | |
| layer rec 'lstm2_bw' #: 1024 | |
| layer rec 'lstm2_fw' #: 1024 | |
| layer pool 'lstm2_pool' #: 2048 | |
| layer rec 'lstm3_bw' #: 1024 | |
| layer rec 'lstm3_fw' #: 1024 | |
| layer pool 'lstm3_pool' #: 2048 | |
| layer rec 'lstm4_bw' #: 1024 | |
| layer rec 'lstm4_fw' #: 1024 | |
| layer pool 'lstm4_pool' #: 2048 | |
| layer rec 'lstm5_bw' #: 1024 | |
| layer rec 'lstm5_fw' #: 1024 | |
| layer rec 'output' #: 10025 | |
| layer eval 'source' #: 40 | |
| net params #: 187862156 | |
| net trainable params: [<tf.Variable 'ctc/W:0' shape=(2048, 10026) dtype=float32_ref>, <tf.Variable 'ctc/b:0' shape=(10026,) dtype=float32_ref>, <tf.Variable 'enc_ctx/W:0' shape=(2048, 1024) dtype=float32_ref>, <tf.Variable 'enc_ctx/b:0' shape=(1024,) dtype=float32_ref>, <tf.Variable 'inv_fertility/W:0' shape=(2048, 1) dtype=float32_ref>, <tf.Variable 'lstm0_bw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref>, <tf.Variable 'lstm0_bw/rec/rnn/basic_lstm_cell/kernel:0' shape=(1064, 4096) dtype=float32_ref>, <tf.Variable 'lstm0_fw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref>, <tf.Variable 'lstm0_fw/rec/rnn/basic_lstm_cell/kernel:0' shape=(1064, 4096) dtype=float32_ref>, <tf.Variable 'lstm1_bw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref>, <tf.Variable 'lstm1_bw/rec/rnn/basic_lstm_cell/kernel:0' shape=(3072, 4096) dtype=float32_ref>, <tf.Variable 'lstm1_fw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref>, <tf.Variable 'lstm1_fw/rec/rnn/basic_lstm_cell/kernel:0' shape=(3072, 4096) dtype=float32_ref>, <tf.Variable 'lstm2_bw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref>, <tf.Variable 'lstm2_bw/rec/rnn/basic_lstm_cell/kernel:0' shape=(3072, 4096) dtype=float32_ref>, <tf.Variable 'lstm2_fw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref>, <tf.Variable 'lstm2_fw/rec/rnn/basic_lstm_cell/kernel:0' shape=(3072, 4096) dtype=float32_ref>, <tf.Variable 'lstm3_bw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref>, <tf.Variable 'lstm3_bw/rec/rnn/basic_lstm_cell/kernel:0' shape=(3072, 4096) dtype=float32_ref>, <tf.Variable 'lstm3_fw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref>, <tf.Variable 'lstm3_fw/rec/rnn/basic_lstm_cell/kernel:0' shape=(3072, 4096) dtype=float32_ref>, <tf.Variable 'lstm4_bw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref>, <tf.Variable 'lstm4_bw/rec/rnn/basic_lstm_cell/kernel:0' shape=(3072, 4096) dtype=float32_ref>, <tf.Variable 'lstm4_fw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref>, <tf.Variable 'lstm4_fw/rec/rnn/basic_lstm_cell/kernel:0' shape=(3072, 4096) dtype=float32_ref>, <tf.Variable 'lstm5_bw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref>, <tf.Variable 'lstm5_bw/rec/rnn/basic_lstm_cell/kernel:0' shape=(3072, 4096) dtype=float32_ref>, <tf.Variable 'lstm5_fw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref>, <tf.Variable 'lstm5_fw/rec/rnn/basic_lstm_cell/kernel:0' shape=(3072, 4096) dtype=float32_ref>, <tf.Variable 'output/rec/energy/W:0' shape=(1024, 1) dtype=float32_ref>, <tf.Variable 'output/rec/output_prob/W:0' shape=(500, 10025) dtype=float32_ref>, <tf.Variable 'output/rec/output_prob/b:0' shape=(10025,) dtype=float32_ref>, <tf.Variable 'output/rec/readout_in/W:0' shape=(3669, 1000) dtype=float32_ref>, <tf.Variable 'output/rec/readout_in/b:0' shape=(1000,) dtype=float32_ref>, <tf.Variable 'output/rec/s/rec/basic_lstm_cell/bias:0' shape=(4000,) dtype=float32_ref>, <tf.Variable 'output/rec/s/rec/basic_lstm_cell/kernel:0' shape=(3669, 4000) dtype=float32_ref>, <tf.Variable 'output/rec/s_transformed/W:0' shape=(1000, 1024) dtype=float32_ref>, <tf.Variable 'output/rec/target_embed/W:0' shape=(10025, 621) dtype=float32_ref>, <tf.Variable 'output/rec/weight_feedback/W:0' shape=(1, 1024) dtype=float32_ref>] | |
| loading weights from data/exp-returnn/model.180 | |
| 2020-02-05 10:40:16.672205: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key lstm0_bw/rec/rnn/basic_lstm_cell/bias not found in checkpoint | |
| load_params_from_file: some variables not found | |
| Variables to restore which are not in checkpoint: ['lstm0_bw/rec/rnn/basic_lstm_cell/bias', 'lstm0_bw/rec/rnn/basic_lstm_cell/kernel', 'lstm0_fw/rec/rnn/basic_lstm_cell/bias', 'lstm0_fw/rec/rnn/basic_lstm_cell/kernel', 'lstm1_bw/rec/rnn/basic_lstm_cell/bias', 'lstm1_bw/rec/rnn/basic_lstm_cell/kernel', 'lstm1_fw/rec/rnn/basic_lstm_cell/bias', 'lstm1_fw/rec/rnn/basic_lstm_cell/kernel', 'lstm2_bw/rec/rnn/basic_lstm_cell/bias', 'lstm2_bw/rec/rnn/basic_lstm_cell/kernel', 'lstm2_fw/rec/rnn/basic_lstm_cell/bias', 'lstm2_fw/rec/rnn/basic_lstm_cell/kernel', 'lstm3_bw/rec/rnn/basic_lstm_cell/bias', 'lstm3_bw/rec/rnn/basic_lstm_cell/kernel', 'lstm3_fw/rec/rnn/basic_lstm_cell/bias', 'lstm3_fw/rec/rnn/basic_lstm_cell/kernel', 'lstm4_bw/rec/rnn/basic_lstm_cell/bias', 'lstm4_bw/rec/rnn/basic_lstm_cell/kernel', 'lstm4_fw/rec/rnn/basic_lstm_cell/bias', 'lstm4_fw/rec/rnn/basic_lstm_cell/kernel', 'lstm5_bw/rec/rnn/basic_lstm_cell/bias', 'lstm5_bw/rec/rnn/basic_lstm_cell/kernel', 'lstm5_fw/rec/rnn/basic_lstm_cell/bias', 'lstm5_fw/rec/rnn/basic_lstm_cell/kernel', 'output/rec/s/rec/basic_lstm_cell/bias', 'output/rec/s/rec/basic_lstm_cell/kernel'] | |
| Could not find mappings for these variables: ['lstm0_bw/rec/rnn/basic_lstm_cell/bias', 'lstm0_bw/rec/rnn/basic_lstm_cell/kernel', 'lstm0_fw/rec/rnn/basic_lstm_cell/bias', 'lstm0_fw/rec/rnn/basic_lstm_cell/kernel', 'lstm1_bw/rec/rnn/basic_lstm_cell/bias', 'lstm1_bw/rec/rnn/basic_lstm_cell/kernel', 'lstm1_fw/rec/rnn/basic_lstm_cell/bias', 'lstm1_fw/rec/rnn/basic_lstm_cell/kernel', 'lstm2_bw/rec/rnn/basic_lstm_cell/bias', 'lstm2_bw/rec/rnn/basic_lstm_cell/kernel', 'lstm2_fw/rec/rnn/basic_lstm_cell/bias', 'lstm2_fw/rec/rnn/basic_lstm_cell/kernel', 'lstm3_bw/rec/rnn/basic_lstm_cell/bias', 'lstm3_bw/rec/rnn/basic_lstm_cell/kernel', 'lstm3_fw/rec/rnn/basic_lstm_cell/bias', 'lstm3_fw/rec/rnn/basic_lstm_cell/kernel', 'lstm4_bw/rec/rnn/basic_lstm_cell/bias', 'lstm4_bw/rec/rnn/basic_lstm_cell/kernel', 'lstm4_fw/rec/rnn/basic_lstm_cell/bias', 'lstm4_fw/rec/rnn/basic_lstm_cell/kernel', 'lstm5_bw/rec/rnn/basic_lstm_cell/bias', 'lstm5_bw/rec/rnn/basic_lstm_cell/kernel', 'lstm5_fw/rec/rnn/basic_lstm_cell/bias', 'lstm5_fw/rec/rnn/basic_lstm_cell/kernel', 'output/rec/s/rec/basic_lstm_cell/bias', 'output/rec/s/rec/basic_lstm_cell/kernel'] var_name_map: {} | |
| All variables in checkpoint: | |
| ctc/W (DT_FLOAT) [2048,10026] | |
| ctc/b (DT_FLOAT) [10026] | |
| enc_ctx/W (DT_FLOAT) [2048,1024] | |
| enc_ctx/b (DT_FLOAT) [1024] | |
| global_step (DT_INT64) [] | |
| inv_fertility/W (DT_FLOAT) [2048,1] | |
| lstm0_bw/rec/W (DT_FLOAT) [40,4096] | |
| lstm0_bw/rec/W_re (DT_FLOAT) [1024,4096] | |
| lstm0_bw/rec/b (DT_FLOAT) [4096] | |
| lstm0_fw/rec/W (DT_FLOAT) [40,4096] | |
| lstm0_fw/rec/W_re (DT_FLOAT) [1024,4096] | |
| lstm0_fw/rec/b (DT_FLOAT) [4096] | |
| lstm1_bw/rec/W (DT_FLOAT) [2048,4096] | |
| lstm1_bw/rec/W_re (DT_FLOAT) [1024,4096] | |
| lstm1_bw/rec/b (DT_FLOAT) [4096] | |
| lstm1_fw/rec/W (DT_FLOAT) [2048,4096] | |
| lstm1_fw/rec/W_re (DT_FLOAT) [1024,4096] | |
| lstm1_fw/rec/b (DT_FLOAT) [4096] | |
| lstm2_bw/rec/W (DT_FLOAT) [2048,4096] | |
| lstm2_bw/rec/W_re (DT_FLOAT) [1024,4096] | |
| lstm2_bw/rec/b (DT_FLOAT) [4096] | |
| lstm2_fw/rec/W (DT_FLOAT) [2048,4096] | |
| lstm2_fw/rec/W_re (DT_FLOAT) [1024,4096] | |
| lstm2_fw/rec/b (DT_FLOAT) [4096] | |
| lstm3_bw/rec/W (DT_FLOAT) [2048,4096] | |
| lstm3_bw/rec/W_re (DT_FLOAT) [1024,4096] | |
| lstm3_bw/rec/b (DT_FLOAT) [4096] | |
| lstm3_fw/rec/W (DT_FLOAT) [2048,4096] | |
| lstm3_fw/rec/W_re (DT_FLOAT) [1024,4096] | |
| lstm3_fw/rec/b (DT_FLOAT) [4096] | |
| lstm4_bw/rec/W (DT_FLOAT) [2048,4096] | |
| lstm4_bw/rec/W_re (DT_FLOAT) [1024,4096] | |
| lstm4_bw/rec/b (DT_FLOAT) [4096] | |
| lstm4_fw/rec/W (DT_FLOAT) [2048,4096] | |
| lstm4_fw/rec/W_re (DT_FLOAT) [1024,4096] | |
| lstm4_fw/rec/b (DT_FLOAT) [4096] | |
| lstm5_bw/rec/W (DT_FLOAT) [2048,4096] | |
| lstm5_bw/rec/W_re (DT_FLOAT) [1024,4096] | |
| lstm5_bw/rec/b (DT_FLOAT) [4096] | |
| lstm5_fw/rec/W (DT_FLOAT) [2048,4096] | |
| lstm5_fw/rec/W_re (DT_FLOAT) [1024,4096] | |
| lstm5_fw/rec/b (DT_FLOAT) [4096] | |
| output/rec/energy/W (DT_FLOAT) [1024,1] | |
| output/rec/output_prob/W (DT_FLOAT) [500,10025] | |
| output/rec/output_prob/b (DT_FLOAT) [10025] | |
| output/rec/readout_in/W (DT_FLOAT) [3669,1000] | |
| output/rec/readout_in/b (DT_FLOAT) [1000] | |
| output/rec/s/rec/lstm_cell/bias (DT_FLOAT) [4000] | |
| output/rec/s/rec/lstm_cell/kernel (DT_FLOAT) [3669,4000] | |
| output/rec/s_transformed/W (DT_FLOAT) [1000,1024] | |
| output/rec/target_embed/W (DT_FLOAT) [10025,621] | |
| output/rec/weight_feedback/W (DT_FLOAT) [1,1024] | |
| All variables to restore: | |
| <tf.Variable 'ctc/W:0' shape=(2048, 10026) dtype=float32_ref> | |
| <tf.Variable 'ctc/b:0' shape=(10026,) dtype=float32_ref> | |
| <tf.Variable 'enc_ctx/W:0' shape=(2048, 1024) dtype=float32_ref> | |
| <tf.Variable 'enc_ctx/b:0' shape=(1024,) dtype=float32_ref> | |
| <tf.Variable 'inv_fertility/W:0' shape=(2048, 1) dtype=float32_ref> | |
| <tf.Variable 'lstm0_bw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref> | |
| <tf.Variable 'lstm0_bw/rec/rnn/basic_lstm_cell/kernel:0' shape=(1064, 4096) dtype=float32_ref> | |
| <tf.Variable 'lstm0_fw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref> | |
| <tf.Variable 'lstm0_fw/rec/rnn/basic_lstm_cell/kernel:0' shape=(1064, 4096) dtype=float32_ref> | |
| <tf.Variable 'lstm1_bw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref> | |
| <tf.Variable 'lstm1_bw/rec/rnn/basic_lstm_cell/kernel:0' shape=(3072, 4096) dtype=float32_ref> | |
| <tf.Variable 'lstm1_fw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref> | |
| <tf.Variable 'lstm1_fw/rec/rnn/basic_lstm_cell/kernel:0' shape=(3072, 4096) dtype=float32_ref> | |
| <tf.Variable 'lstm2_bw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref> | |
| <tf.Variable 'lstm2_bw/rec/rnn/basic_lstm_cell/kernel:0' shape=(3072, 4096) dtype=float32_ref> | |
| <tf.Variable 'lstm2_fw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref> | |
| <tf.Variable 'lstm2_fw/rec/rnn/basic_lstm_cell/kernel:0' shape=(3072, 4096) dtype=float32_ref> | |
| <tf.Variable 'lstm3_bw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref> | |
| <tf.Variable 'lstm3_bw/rec/rnn/basic_lstm_cell/kernel:0' shape=(3072, 4096) dtype=float32_ref> | |
| <tf.Variable 'lstm3_fw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref> | |
| <tf.Variable 'lstm3_fw/rec/rnn/basic_lstm_cell/kernel:0' shape=(3072, 4096) dtype=float32_ref> | |
| <tf.Variable 'lstm4_bw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref> | |
| <tf.Variable 'lstm4_bw/rec/rnn/basic_lstm_cell/kernel:0' shape=(3072, 4096) dtype=float32_ref> | |
| <tf.Variable 'lstm4_fw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref> | |
| <tf.Variable 'lstm4_fw/rec/rnn/basic_lstm_cell/kernel:0' shape=(3072, 4096) dtype=float32_ref> | |
| <tf.Variable 'lstm5_bw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref> | |
| <tf.Variable 'lstm5_bw/rec/rnn/basic_lstm_cell/kernel:0' shape=(3072, 4096) dtype=float32_ref> | |
| <tf.Variable 'lstm5_fw/rec/rnn/basic_lstm_cell/bias:0' shape=(4096,) dtype=float32_ref> | |
| <tf.Variable 'lstm5_fw/rec/rnn/basic_lstm_cell/kernel:0' shape=(3072, 4096) dtype=float32_ref> | |
| <tf.Variable 'output/rec/energy/W:0' shape=(1024, 1) dtype=float32_ref> | |
| <tf.Variable 'output/rec/output_prob/W:0' shape=(500, 10025) dtype=float32_ref> | |
| <tf.Variable 'output/rec/output_prob/b:0' shape=(10025,) dtype=float32_ref> | |
| <tf.Variable 'output/rec/readout_in/W:0' shape=(3669, 1000) dtype=float32_ref> | |
| <tf.Variable 'output/rec/readout_in/b:0' shape=(1000,) dtype=float32_ref> | |
| <tf.Variable 'output/rec/s/rec/basic_lstm_cell/bias:0' shape=(4000,) dtype=float32_ref> | |
| <tf.Variable 'output/rec/s/rec/basic_lstm_cell/kernel:0' shape=(3669, 4000) dtype=float32_ref> | |
| <tf.Variable 'output/rec/s_transformed/W:0' shape=(1000, 1024) dtype=float32_ref> | |
| <tf.Variable 'output/rec/target_embed/W:0' shape=(10025, 621) dtype=float32_ref> | |
| <tf.Variable 'output/rec/weight_feedback/W:0' shape=(1, 1024) dtype=float32_ref> | |
| <tf.Variable 'global_step:0' shape=() dtype=int64_ref> | |
| Variables to restore which are not in checkpoint: | |
| lstm0_bw/rec/rnn/basic_lstm_cell/bias | |
| lstm0_bw/rec/rnn/basic_lstm_cell/kernel | |
| lstm0_fw/rec/rnn/basic_lstm_cell/bias | |
| lstm0_fw/rec/rnn/basic_lstm_cell/kernel | |
| lstm1_bw/rec/rnn/basic_lstm_cell/bias | |
| lstm1_bw/rec/rnn/basic_lstm_cell/kernel | |
| lstm1_fw/rec/rnn/basic_lstm_cell/bias | |
| lstm1_fw/rec/rnn/basic_lstm_cell/kernel | |
| lstm2_bw/rec/rnn/basic_lstm_cell/bias | |
| lstm2_bw/rec/rnn/basic_lstm_cell/kernel | |
| lstm2_fw/rec/rnn/basic_lstm_cell/bias | |
| lstm2_fw/rec/rnn/basic_lstm_cell/kernel | |
| lstm3_bw/rec/rnn/basic_lstm_cell/bias | |
| lstm3_bw/rec/rnn/basic_lstm_cell/kernel | |
| lstm3_fw/rec/rnn/basic_lstm_cell/bias | |
| lstm3_fw/rec/rnn/basic_lstm_cell/kernel | |
| lstm4_bw/rec/rnn/basic_lstm_cell/bias | |
| lstm4_bw/rec/rnn/basic_lstm_cell/kernel | |
| lstm4_fw/rec/rnn/basic_lstm_cell/bias | |
| lstm4_fw/rec/rnn/basic_lstm_cell/kernel | |
| lstm5_bw/rec/rnn/basic_lstm_cell/bias | |
| lstm5_bw/rec/rnn/basic_lstm_cell/kernel | |
| lstm5_fw/rec/rnn/basic_lstm_cell/bias | |
| lstm5_fw/rec/rnn/basic_lstm_cell/kernel | |
| output/rec/s/rec/basic_lstm_cell/bias | |
| output/rec/s/rec/basic_lstm_cell/kernel | |
| Variables in checkpoint which are not needed for restore: | |
| lstm0_bw/rec/W | |
| lstm0_bw/rec/W_re | |
| lstm0_bw/rec/b | |
| lstm0_fw/rec/W | |
| lstm0_fw/rec/W_re | |
| lstm0_fw/rec/b | |
| lstm1_bw/rec/W | |
| lstm1_bw/rec/W_re | |
| lstm1_bw/rec/b | |
| lstm1_fw/rec/W | |
| lstm1_fw/rec/W_re | |
| lstm1_fw/rec/b | |
| lstm2_bw/rec/W | |
| lstm2_bw/rec/W_re | |
| lstm2_bw/rec/b | |
| lstm2_fw/rec/W | |
| lstm2_fw/rec/W_re | |
| lstm2_fw/rec/b | |
| lstm3_bw/rec/W | |
| lstm3_bw/rec/W_re | |
| lstm3_bw/rec/b | |
| lstm3_fw/rec/W | |
| lstm3_fw/rec/W_re | |
| lstm3_fw/rec/b | |
| lstm4_bw/rec/W | |
| lstm4_bw/rec/W_re | |
| lstm4_bw/rec/b | |
| lstm4_fw/rec/W | |
| lstm4_fw/rec/W_re | |
| lstm4_fw/rec/b | |
| lstm5_bw/rec/W | |
| lstm5_bw/rec/W_re | |
| lstm5_bw/rec/b | |
| lstm5_fw/rec/W | |
| lstm5_fw/rec/W_re | |
| lstm5_fw/rec/b | |
| output/rec/s/rec/lstm_cell/bias | |
| output/rec/s/rec/lstm_cell/kernel | |
| Probably we can restore these: | |
| (None) | |
| Error, some entry is missing in the checkpoint 'data/exp-returnn/model.180': <class 'tensorflow.python.framework.errors_impl.NotFoundError'>: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error: | |
| Key lstm0_bw/rec/rnn/basic_lstm_cell/bias not found in checkpoint | |
| [[node saver/save/RestoreV2 (defined at /home/ubuntu/manish/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py:1377) ]] | |
| Caused by op 'saver/save/RestoreV2', defined at: | |
| File "returnn/rnn.py", line 654, in <module> | |
| main(sys.argv) | |
| File "returnn/rnn.py", line 642, in main | |
| execute_main_task() | |
| File "returnn/rnn.py", line 569, in execute_main_task | |
| engine.init_train_from_config(config, train_data, dev_data, eval_data) | |
| File "/home/ubuntu/manish/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFEngine.py", line 891, in init_train_from_config | |
| self.init_network_from_config(config) | |
| File "/home/ubuntu/manish/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFEngine.py", line 972, in init_network_from_config | |
| self.network.load_params_from_file(model_epoch_filename, session=self.tf_session) | |
| File "/home/ubuntu/manish/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 1425, in load_params_from_file | |
| self._create_saver() | |
| File "/home/ubuntu/manish/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py", line 1377, in _create_saver | |
| var_list=self.get_saveable_params_list(), max_to_keep=2 ** 31 - 1) | |
| File "/home/ubuntu/tf1.13.1/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 832, in __init__ | |
| self.build() | |
| File "/home/ubuntu/tf1.13.1/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 844, in build | |
| self._build(self._filename, build_save=True, build_restore=True) | |
| File "/home/ubuntu/tf1.13.1/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 881, in _build | |
| build_save=build_save, build_restore=build_restore) | |
| File "/home/ubuntu/tf1.13.1/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 513, in _build_internal | |
| restore_sequentially, reshape) | |
| File "/home/ubuntu/tf1.13.1/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 332, in _AddRestoreOps | |
| restore_sequentially) | |
| File "/home/ubuntu/tf1.13.1/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 580, in bulk_restore | |
| return io_ops.restore_v2(filename_tensor, names, slices, dtypes) | |
| File "/home/ubuntu/tf1.13.1/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1572, in restore_v2 | |
| name=name) | |
| File "/home/ubuntu/tf1.13.1/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper | |
| op_def=op_def) | |
| File "/home/ubuntu/tf1.13.1/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func | |
| return func(*args, **kwargs) | |
| File "/home/ubuntu/tf1.13.1/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op | |
| op_def=op_def) | |
| File "/home/ubuntu/tf1.13.1/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__ | |
| self._traceback = tf_stack.extract_stack() | |
| NotFoundError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error: | |
| Key lstm0_bw/rec/rnn/basic_lstm_cell/bias not found in checkpoint | |
| [[node saver/save/RestoreV2 (defined at /home/ubuntu/manish/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetwork.py:1377) ]] | |
| CustomCheckpointLoader was not able to recover. | |
| Exiting now because model cannot be loaded. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment