marian-bad-statistics.log

[2022-07-04 22:03:04] [marian] Marian v1.11.0 f00d0621 2022-02-08 08:39:24 -0800
[2022-07-04 22:03:04] [marian] Running on 104-171-200-250 as process 1727207 with command line:
[2022-07-04 22:03:04] [marian] /home/ubuntu/marian/build/marian --model /home/ubuntu/stash/fdi-6+6-8-1024-4096-8000-0.0001-0.1/fr-xx-r42//mode>
[2022-07-04 22:03:04] [config] after: 0e
[2022-07-04 22:03:04] [config] after-batches: 0
[2022-07-04 22:03:04] [config] after-epochs: 0
[2022-07-04 22:03:04] [config] all-caps-every: 0
[2022-07-04 22:03:04] [config] allow-unk: false
[2022-07-04 22:03:04] [config] authors: false
[2022-07-04 22:03:04] [config] beam-size: 12
[2022-07-04 22:03:04] [config] bert-class-symbol: "[CLS]"
[2022-07-04 22:03:04] [config] bert-mask-symbol: "[MASK]"
[2022-07-04 22:03:04] [config] bert-masking-fraction: 0.15
[2022-07-04 22:03:04] [config] bert-sep-symbol: "[SEP]"
[2022-07-04 22:03:04] [config] bert-train-type-embeddings: true
[2022-07-04 22:03:04] [config] bert-type-vocab-size: 2
[2022-07-04 22:03:04] [config] build-info: ""
[2022-07-04 22:03:04] [config] check-gradient-nan: false
[2022-07-04 22:03:04] [config] check-nan: false
[2022-07-04 22:03:04] [config] cite: false
[2022-07-04 22:03:04] [config] clip-norm: 5
[2022-07-04 22:03:04] [config] cost-scaling:
[2022-07-04 22:03:04] [config]   []
[2022-07-04 22:03:04] [config] cost-type: ce-mean-words
[2022-07-04 22:03:04] [config] cpu-threads: 0
[2022-07-04 22:03:04] [config] data-threads: 8
[2022-07-04 22:03:04] [config] data-weighting: ""
[2022-07-04 22:03:04] [config] data-weighting-type: sentence
[2022-07-04 22:03:04] [config] dec-cell: gru
[2022-07-04 22:03:04] [config] dec-cell-base-depth: 2
[2022-07-04 22:03:04] [config] dec-cell-high-depth: 1
[2022-07-04 22:03:04] [config] dec-depth: 6
[2022-07-04 22:03:04] [config] devices:
[2022-07-04 22:03:04] [config]   - 0
[2022-07-04 22:03:04] [config] dim-emb: 1024
[2022-07-04 22:03:04] [config] dim-rnn: 1024
[2022-07-04 22:03:04] [config] dim-vocabs:
[2022-07-04 22:03:04] [config]   - 8000
[2022-07-04 22:03:04] [config]   - 8000
[2022-07-04 22:03:04] [config] disp-first: 0
[2022-07-04 22:03:04] [config] disp-freq: 00
[2022-07-04 22:03:04] [config] disp-label-counts: true
[2022-07-04 22:03:04] [config] dropout-rnn: 0
[2022-07-04 22:03:04] [config] dropout-src: 0
2022-07-04 22:03:04] [config] dec-depth: 6
[2022-07-04 22:03:04] [config] devices:
[2022-07-04 22:03:04] [config]   - 0
[2022-07-04 22:03:04] [config] dim-emb: 1024
[2022-07-04 22:03:04] [config] dim-rnn: 1024
[2022-07-04 22:03:04] [config] dim-vocabs:
[2022-07-04 22:03:04] [config]   - 8000
[2022-07-04 22:03:04] [config]   - 8000
[2022-07-04 22:03:04] [config] disp-first: 0
[2022-07-04 22:03:04] [config] disp-freq: 00
[2022-07-04 22:03:04] [config] disp-label-counts: true
[2022-07-04 22:03:04] [config] dropout-rnn: 0
[2022-07-04 22:03:04] [config] dropout-src: 0
[2022-07-04 22:03:04] [config] dropout-trg: 0
[2022-07-04 22:03:04] [config] dump-config: ""
[2022-07-04 22:03:04] [config] dynamic-gradient-scaling:
[2022-07-04 22:03:04] [config]   []
[2022-07-04 22:03:04] [config] early-stopping: 5
[2022-07-04 22:03:04] [config] early-stopping-on: first
[2022-07-04 22:03:04] [config] embedding-fix-src: false
[2022-07-04 22:03:04] [config] embedding-fix-trg: false
[2022-07-04 22:03:04] [config] embedding-normalization: false
[2022-07-04 22:03:04] [config] embedding-vectors:
[2022-07-04 22:03:04] [config]   []
[2022-07-04 22:03:04] [config] enc-cell: gru
[2022-07-04 22:03:04] [config] enc-cell-depth: 1
[2022-07-04 22:03:04] [config] enc-depth: 6
[2022-07-04 22:03:04] [config] enc-type: bidirectional
[2022-07-04 22:03:04] [config] english-title-case-every: 0
[2022-07-04 22:03:04] [config] exponential-smoothing: 0.0001
[2022-07-04 22:03:04] [config] factor-weight: 1
[2022-07-04 22:03:04] [config] factors-combine: sum
[2022-07-04 22:03:04] [config] factors-dim-emb: 0
[2022-07-04 22:03:04] [config] gradient-checkpointing: false
[2022-07-04 22:03:04] [config] gradient-norm-average-window: 100
[2022-07-04 22:03:04] [config] guided-alignment: none
[2022-07-04 22:03:04] [config] guided-alignment-cost: mse
[2022-07-04 22:03:04] [config] guided-alignment-weight: 0.1
[2022-07-04 22:03:04] [config] ignore-model-config: false
2022-07-04 22:03:04] [config] input-types:
[2022-07-04 22:03:04] [config]   []
[2022-07-04 22:03:04] [config] interpolate-env-vars: false
[2022-07-04 22:03:04] [config] keep-best: true
[2022-07-04 22:03:04] [config] label-smoothing: 0.1
[2022-07-04 22:03:04] [config] layer-normalization: false
[2022-07-04 22:03:04] [config] learn-rate: 0.0001
[2022-07-04 22:03:04] [config] lemma-dependency: ""
[2022-07-04 22:03:04] [config] lemma-dim-emb: 0
[2022-07-04 22:03:04] [config] log: /home/ubuntu/stash/fdi-6+6-8-1024-4096-8000-0.0001-0.1/fr-xx-r42//train.log
[2022-07-04 22:03:04] [config] log-level: info
[2022-07-04 22:03:04] [config] log-time-zone: ""
[2022-07-04 22:03:04] [config] logical-epoch:
[2022-07-04 22:03:04] [config]   - 1e
[2022-07-04 22:03:04] [config]   - 0
[2022-07-04 22:03:04] [config] lr-decay: 0
[2022-07-04 22:03:04] [config] lr-decay-freq: 50000
[2022-07-04 22:03:04] [config] lr-decay-inv-sqrt:
[2022-07-04 22:03:04] [config]   - 8000
[2022-07-04 22:03:04] [config] lr-decay-repeat-warmup: false
[2022-07-04 22:03:04] [config] lr-decay-reset-optimizer: false
[2022-07-04 22:03:04] [config] lr-decay-start:
[2022-07-04 22:03:04] [config]   - 10
[2022-07-04 22:03:04] [config]   - 1
[2022-07-04 22:03:04] [config] lr-decay-strategy: epoch+stalled
[2022-07-04 22:03:04] [config] lr-report: true
[2022-07-04 22:03:04] [config] lr-warmup: 8000
[2022-07-04 22:03:04] [config] lr-warmup-at-reload: false
[2022-07-04 22:03:04] [config] lr-warmup-cycle: false
[2022-07-04 22:03:04] [config] lr-warmup-start-rate: 0
[2022-07-04 22:03:04] [config] max-length: 5000
[2022-07-04 22:03:04] [config] max-length-crop: true
[2022-07-04 22:03:04] [config] max-length-factor: 3
[2022-07-04 22:03:04] [config] maxi-batch: 100
[2022-07-04 22:03:04] [config] maxi-batch-sort: trg
[2022-07-04 22:03:04] [config] mini-batch: 64
[2022-07-04 22:03:04] [config] mini-batch-fit: true
[2022-07-04 22:03:04] [config] mini-batch-fit-step: 10
[2022-07-04 22:03:04] [config] mini-batch-round-up: true
[2022-07-04 22:03:04] [config] mini-batch-track-lr: false
[2022-07-04 22:03:04] [config] mini-batch-warmup: 0
[2022-07-04 22:03:04] [config] model: /home/ubuntu/stash/fdi-6+6-8-1024-4096-8000-0.0001-0.1/fr-xx-r42//model.npz
[2022-07-04 22:03:04] [config] multi-loss-type: sum
[2022-07-04 22:03:04] [config] n-best: false
[2022-07-04 22:03:04] [config] no-nccl: false
[2022-07-04 22:03:04] [config] no-reload: false
[2022-07-04 22:03:04] [config] no-restore-corpus: false
[2022-07-04 22:03:04] [config] normalize: 0.6
[2022-07-04 22:03:04] [config] normalize-gradient: false
[2022-07-04 22:03:04] [config] num-devices: 0
[2022-07-04 22:03:04] [config] optimizer: adam
[2022-07-04 22:03:04] [config] optimizer-delay: 1
[2022-07-04 22:03:04] [config] optimizer-params:
[2022-07-04 22:03:04] [config]   - 0.9
[2022-07-04 22:03:04] [config]   - 0.98
[2022-07-04 22:03:04] [config]   - 1e-09
[2022-07-04 22:03:04] [config] output-omit-bias: false
[2022-07-04 22:03:04] [config] overwrite: false
[2022-07-04 22:03:04] [config] precision:
[2022-07-04 22:03:04] [config]   - float32
[2022-07-04 22:03:04] [config]   - float32
[2022-07-04 22:03:04] [config] pretrained-model: ""
[2022-07-04 22:03:04] [config] quantize-biases: false
[2022-07-04 22:03:04] [config] quantize-bits: 0
[2022-07-04 22:03:04] [config] quantize-log-based: false
[2022-07-04 22:03:04] [config] quantize-optimization-steps: 0
[2022-07-04 22:03:04] [config] quiet: false
[2022-07-04 22:03:04] [config] quiet-translation: true
[2022-07-04 22:03:04] [config] relative-paths: false
[2022-07-04 22:03:04] [config] right-left: false
[2022-07-04 22:03:04] [config] save-freq: 500
[2022-07-04 22:03:04] [config] seed: 42
[2022-07-04 22:03:04] [config] sentencepiece-alphas:
[2022-07-04 22:03:04] [config]   []
[2022-07-04 22:03:04] [config] sentencepiece-max-lines: 2000000
[2022-07-04 22:03:04] [config] sentencepiece-options: --character_coverage=1.0 --user_defined_symbols=BE,CA,CH,FR
[2022-07-04 22:03:04] [config] sharding: global
[2022-07-04 22:03:04] [config] shuffle: data
[2022-07-04 22:03:04] [config] shuffle-in-ram: true
[2022-07-04 22:03:04] [config] sigterm: save-and-exit
[2022-07-04 22:03:04] [config] skip: false
[2022-07-04 22:03:04] [config] sqlite: ""
[2022-07-04 22:03:04] [config] sqlite-drop: false
[2022-07-04 22:03:04] [config] sync-freq: 200u
[2022-07-04 22:03:04] [config] sync-sgd: true
[2022-07-04 22:03:04] [config] tempdir: /tmp
[2022-07-04 22:03:04] [config] tied-embeddings: false
[2022-07-04 22:03:04] [config] tied-embeddings-all: true
[2022-07-04 22:03:04] [config] tied-embeddings-src: false
[2022-07-04 22:03:04] [config] train-embedder-rank:
[2022-07-04 22:03:04] [config]   []
[2022-07-04 22:03:04] [config] train-sets:
[2022-07-04 22:03:04] [config]   - /home/ubuntu/stash/fdi-data/train.fr-xx.fr
[2022-07-04 22:03:04] [config]   - /home/ubuntu/stash/fdi-data/train.fr-xx.xx
[2022-07-04 22:03:04] [config] transformer-aan-activation: swish
[2022-07-04 22:03:04] [config] transformer-aan-depth: 2
[2022-07-04 22:03:04] [config] transformer-aan-nogate: false
[2022-07-04 22:03:04] [config] transformer-decoder-autoreg: self-attention
[2022-07-04 22:03:04] [config] transformer-decoder-dim-ffn: 0
[2022-07-04 22:03:04] [config] transformer-decoder-ffn-depth: 0
[2022-07-04 22:03:04] [config] transformer-depth-scaling: false
[2022-07-04 22:03:04] [config] transformer-dim-aan: 2048
[2022-07-04 22:03:04] [config] transformer-dim-ffn: 4096
[2022-07-04 22:03:04] [config] transformer-dropout: 0.1
[2022-07-04 22:03:04] [config] transformer-dropout-attention: 0.1
[2022-07-04 22:03:04] [config] transformer-dropout-ffn: 0.1
[2022-07-04 22:03:04] [config] transformer-ffn-activation: swish
[2022-07-04 22:03:04] [config] transformer-ffn-depth: 2
[2022-07-04 22:03:04] [config] transformer-guided-alignment-layer: last
[2022-07-04 22:03:04] [config] transformer-heads: 8
[2022-07-04 22:03:04] [config] transformer-no-projection: false
[2022-07-04 22:03:04] [config] transformer-pool: false
[2022-07-04 22:03:04] [config] transformer-postprocess: da
[2022-07-04 22:03:04] [config] transformer-postprocess-emb: d
[2022-07-04 22:03:04] [config] transformer-postprocess-top: ""
[2022-07-04 22:03:04] [config] transformer-preprocess: n
[2022-07-04 22:03:04] [config] transformer-tied-layers:
[2022-07-04 22:03:04] [config]   []
[2022-07-04 22:03:04] [config] transformer-train-position-embeddings: false
[2022-07-04 22:03:04] [config] tsv: false
[2022-07-04 22:03:04] [config] tsv-fields: 0
[2022-07-04 22:03:04] [config] type: transformer
[2022-07-04 22:03:04] [config] ulr: false
[2022-07-04 22:03:04] [config] ulr-dim-emb: 0
[2022-07-04 22:03:04] [config] ulr-dropout: 0
[2022-07-04 22:03:04] [config] ulr-keys-vectors: ""
[2022-07-04 22:03:04] [config] ulr-query-vectors: ""
[2022-07-04 22:03:04] [config] ulr-softmax-temperature: 1
[2022-07-04 22:03:04] [config] ulr-trainable-transformation: false
[2022-07-04 22:03:04] [config] unlikelihood-loss: false
[2022-07-04 22:03:04] [config] valid-freq: 500
[2022-07-04 22:03:04] [config] valid-log: /home/ubuntu/stash/fdi-6+6-8-1024-4096-8000-0.0001-0.1/fr-xx-r42//valid.log
[2022-07-04 22:03:04] [config] valid-max-length: 5000
[2022-07-04 22:03:04] [config] valid-metrics:
[2022-07-04 22:03:04] [config]   - ce-mean-words
[2022-07-04 22:03:04] [config]   - perplexity
[2022-07-04 22:03:04] [config] valid-mini-batch: 16
[2022-07-04 22:03:04] [config] valid-reset-stalled: false
[2022-07-04 22:03:04] [config] valid-script-args:
[2022-07-04 22:03:04] [config]   []
[2022-07-04 22:03:04] [config] valid-script-path: ""
[2022-07-04 22:03:04] [config] valid-sets:
[2022-07-04 22:03:04] [config]   - /home/ubuntu/stash/fdi-data/valid.fr-xx.fr
[2022-07-04 22:03:04] [config]   - /home/ubuntu/stash/fdi-data/valid.fr-xx.xx
[2022-07-04 22:03:04] [config] valid-translation-output: ""
[2022-07-04 22:03:04] [config] vocabs:
[2022-07-04 22:03:04] [config]   - /home/ubuntu/stash/fdi-6+6-8-1024-4096-8000-0.0001-0.1/fr-xx-r42//vocab.src.spm
[2022-07-04 22:03:04] [config]   - /home/ubuntu/stash/fdi-6+6-8-1024-4096-8000-0.0001-0.1/fr-xx-r42//vocab.src.spm
[2022-07-04 22:03:04] [config] word-penalty: 0
[2022-07-04 22:03:04] [config] word-scores: false
[2022-07-04 22:03:04] [config] workspace: 10185
[2022-07-04 22:03:04] [config] Model is being created with Marian v1.11.0 f00d0621 2022-02-08 08:39:24 -0800
[2022-07-04 22:03:04] Using synchronous SGD
[2022-07-04 22:03:04] [comm] Compiled without MPI support. Running as a single process on 104-171-200-250
[2022-07-04 22:03:04] Synced seed 42
[2022-07-04 22:03:04] [data] Loading SentencePiece vocabulary from file /home/ubuntu/stash/fdi-6+6-8-1024-4096-8000-0.0001-0.1/fr-xx-r42//voca>
[2022-07-04 22:03:04] [data] Setting vocabulary size for input 0 to 8,000
[2022-07-04 22:03:04] [data] Loading SentencePiece vocabulary from file /home/ubuntu/stash/fdi-6+6-8-1024-4096-8000-0.0001-0.1/fr-xx-r42//voca>
[2022-07-04 22:03:04] [data] Setting vocabulary size for input 1 to 8,000
[2022-07-04 22:03:04] [batching] Collecting statistics for batch fitting with step size 10
[2022-07-04 22:03:05] [memory] Extending reserved space to 10240 MB (device gpu0)
[2022-07-04 22:03:05] [comm] Using NCCL 2.8.3 for GPU communication
[2022-07-04 22:03:05] [comm] Using global sharding
[2022-07-04 22:03:05] [comm] NCCLCommunicators constructed successfully
[2022-07-04 22:03:05] [training] Using 1 GPUs
[2022-07-04 22:03:05] [logits] Applying loss function for 1 factor(s)
[2022-07-04 22:03:05] [memory] Reserving 704 MB, device gpu0
[2022-07-04 22:03:05] [gpu] 16-bit TensorCores enabled for float32 matrix operations
[2022-07-04 22:03:05] [memory] Reserving 704 MB, device gpu0
[2022-07-04 22:04:20] [batching] Done. Typical MB size is 2,350 target words
[2022-07-04 22:04:20] [memory] Extending reserved space to 10240 MB (device gpu0)
[2022-07-04 22:04:20] [comm] Using NCCL 2.8.3 for GPU communication
[2022-07-04 22:04:20] [comm] Using global sharding
[2022-07-04 22:04:20] [comm] NCCLCommunicators constructed successfully
[2022-07-04 22:04:20] [training] Using 1 GPUs

[2022-07-04 22:04:20] Training started
[2022-07-04 22:04:20] [data] Shuffling data
[2022-07-04 22:04:26] [data] Done reading 358,787 sentences
[2022-07-04 22:04:26] [data] Done shuffling 358,787 sentences (cached in RAM)
[2022-07-04 22:04:26] Error: Missing batch statistics
[2022-07-04 22:04:26] Error: Aborted from size_t marian::data::BatchStats::findBatchSize(const std::vector<long unsigned int>&, marian::data::>

[CALL STACK]
[0x5614255115a7]    marian::data::BatchStats::  findBatchSize  (std::vector<unsigned long,std::allocator<unsigned long>> const&,  std::_Rb_tre>
[0x56142556800c]    marian::data::BatchGenerator<marian::data::CorpusBase>::  fetchBatches  () + 0x181c
[0x5614255689b3]    marian::ThreadPool::enqueue<marian::data::BatchGenerator<marian::data::CorpusBase>::fetchBatchesAsync()::{lambda()#1}>(std>
[0x561425569573]    std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base,std::__future_base::_Result_base::_Deleter> (),std>
[0x5614254963ad]    std::__future_base::_State_baseV2::  _M_do_set  (std::function<std::unique_ptr<std::__future_base::_Result_base,std::__fut>
[0x7f4844fc847f]                                                       + 0x1147f
[0x5614254a1710]    std::__future_base::_Task_state<marian::ThreadPool::enqueue<marian::data::BatchGenerator<marian::data::CorpusBase>::fetchB>
[0x561425497a30]    std::thread::_State_impl<std::thread::_Invoker<std::tuple<marian::ThreadPool::reserve(unsigned long)::{lambda()#1}>>>::  _>
[0x56142779d5b4]                                                       + 0x34745b4
[0x7f4844fbf609]                                                       + 0x8609
[0x7f4844d95163]    clone                                              + 0x43