Created
February 13, 2020 14:34
-
-
Save paolo-losi/be7dd8c825c1367001ef4ad4e0699875 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I Early stop triggered as (for last 4 steps) validation loss: 43.402323 with standard deviation: 0.395598 and mean: 43.838659 | |
I FINISHED optimization in 3:21:50.517237 | |
WARNING:tensorflow:From /home/trainer/ds-train/lib/python3.6/site-packages/tensorflow_core/contrib/rnn/python/ops/lstm_ops.py:597: Layer.add_variable (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. | |
Instructions for updating: | |
Please use `layer.add_weight` method instead. | |
W0213 14:28:47.242244 140350741464896 deprecation.py:323] From /home/trainer/ds-train/lib/python3.6/site-packages/tensorflow_core/contrib/rnn/python/ops/lstm_ops.py:597: Layer.add_variable (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. | |
Instructions for updating: | |
Please use `layer.add_weight` method instead. | |
INFO:tensorflow:Restoring parameters from /mnt/checkpoints/best_dev-8914 | |
I0213 14:28:47.313651 140350741464896 saver.py:1284] Restoring parameters from /mnt/checkpoints/best_dev-8914 | |
I Restored variables from best validation checkpoint at /mnt/checkpoints/best_dev-8914, step 8914 | |
Testing model on /mnt/extracted/data/M-AILABS/it_IT/it_IT_test.csv | |
Test epoch | Steps: 0 | Elapsed Time: 0:00:00 | |
Fatal Python error: Segmentation fault | |
Thread 0x00007fa5414bf700 (most recent call first): | |
File "/usr/lit/train.sh: line 48: 224 Segmentation fault (core dumped) python -u DeepSpeech.py --show_progressbar True --use_cudnn_rnn True --automatic_mixed_precision True --alphabet_config_path /mnt/models/alphabet.txt --lm_binary_path /mnt/lm/lm.binary --lm_trie_path /mnt/lm/trie --feature_cache /mnt/sources/feature_cache --train_files ${all_train_csv} --dev_files ${all_dev_csv} --test_files ${all_test_csv} --train_batch_size ${BATCH_SIZE} --dev_batch_size ${BATCH_SIZE} --test_batch_size ${BATCH_SIZE} --n_hidden ${N_HIDDEN} --epochs ${EPOCHS} --learning_rate ${LEARNING_RATE} --dropout_rate ${DROPOUT} --lm_alpha ${LM_ALPHA} --lm_beta ${LM_BETA} ${EARLY_STOP_FLAG} --checkpoint_dir /mnt/checkpoints/ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment