Skip to content

Instantly share code, notes, and snippets.

@paolo-losi
Created February 13, 2020 14:34
Show Gist options
  • Save paolo-losi/be7dd8c825c1367001ef4ad4e0699875 to your computer and use it in GitHub Desktop.
Save paolo-losi/be7dd8c825c1367001ef4ad4e0699875 to your computer and use it in GitHub Desktop.
I Early stop triggered as (for last 4 steps) validation loss: 43.402323 with standard deviation: 0.395598 and mean: 43.838659
I FINISHED optimization in 3:21:50.517237
WARNING:tensorflow:From /home/trainer/ds-train/lib/python3.6/site-packages/tensorflow_core/contrib/rnn/python/ops/lstm_ops.py:597: Layer.add_variable (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.add_weight` method instead.
W0213 14:28:47.242244 140350741464896 deprecation.py:323] From /home/trainer/ds-train/lib/python3.6/site-packages/tensorflow_core/contrib/rnn/python/ops/lstm_ops.py:597: Layer.add_variable (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.add_weight` method instead.
INFO:tensorflow:Restoring parameters from /mnt/checkpoints/best_dev-8914
I0213 14:28:47.313651 140350741464896 saver.py:1284] Restoring parameters from /mnt/checkpoints/best_dev-8914
I Restored variables from best validation checkpoint at /mnt/checkpoints/best_dev-8914, step 8914
Testing model on /mnt/extracted/data/M-AILABS/it_IT/it_IT_test.csv
Test epoch | Steps: 0 | Elapsed Time: 0:00:00
Fatal Python error: Segmentation fault
Thread 0x00007fa5414bf700 (most recent call first):
File "/usr/lit/train.sh: line 48: 224 Segmentation fault (core dumped) python -u DeepSpeech.py --show_progressbar True --use_cudnn_rnn True --automatic_mixed_precision True --alphabet_config_path /mnt/models/alphabet.txt --lm_binary_path /mnt/lm/lm.binary --lm_trie_path /mnt/lm/trie --feature_cache /mnt/sources/feature_cache --train_files ${all_train_csv} --dev_files ${all_dev_csv} --test_files ${all_test_csv} --train_batch_size ${BATCH_SIZE} --dev_batch_size ${BATCH_SIZE} --test_batch_size ${BATCH_SIZE} --n_hidden ${N_HIDDEN} --epochs ${EPOCHS} --learning_rate ${LEARNING_RATE} --dropout_rate ${DROPOUT} --lm_alpha ${LM_ALPHA} --lm_beta ${LM_BETA} ${EARLY_STOP_FLAG} --checkpoint_dir /mnt/checkpoints/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment