-
-
Save Max-Ryujin/b2da56c72afaf28850dd7384c97a5b2a to your computer and use it in GitHub Desktop.
Freeze in /u/maximilian.kannen/setups/20230406_feat/alias/experiments/switchboard/ctc/feat/train_nn/conformer_bs5k_scf_baseline_big/log.run.1
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| --------------------- Slurm Task Prolog ------------------------ | |
| Job ID: 2873335 | |
| Job name: ReturnnTrainingJob.wQIA3Rc1nLak.run | |
| Host: cn-505 | |
| Date: Di 31. Okt 00:04:38 CET 2023 | |
| User: maximilian.kannen | |
| Slurm account: hlt | |
| Slurm partition: gpu_24gb | |
| Work dir: | |
| ------------------ | |
| Node usage: | |
| JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) | |
| 2841495_1 gpu_24gb i6_core. zeyer R 1-09:57:46 1 cn-505 | |
| 2873335_1 gpu_24gb ReturnnT maximili R 0:01 1 cn-505 | |
| 2873337_1 gpu_24gb ReturnnT maximili R 0:01 1 cn-505 | |
| ------------------ | |
| Show launch script with: | |
| sacct -B -j | |
| ------------------ | |
| --------------------- Slurm Task Prolog ------------------------ | |
| [2023-10-31 00:04:39,351] INFO: [32mGenerating grammar tables from /usr/local/lib/python3.8/dist-packages/blib2to3/Grammar.txt[0m | |
| [2023-10-31 00:04:39,362] INFO: [32mWriting grammar tables to /u/maximilian.kannen/.cache/black/22.3.0/Grammar3.8.10.final.0.pickle[0m | |
| [2023-10-31 00:04:39,362] INFO: [32mWriting failed: [Errno 2] No such file or directory: '/u/maximilian.kannen/.cache/black/22.3.0/tmpxo4a5aig'[0m | |
| [2023-10-31 00:04:39,362] INFO: [32mGenerating grammar tables from /usr/local/lib/python3.8/dist-packages/blib2to3/PatternGrammar.txt[0m | |
| [2023-10-31 00:04:39,364] INFO: [32mWriting grammar tables to /u/maximilian.kannen/.cache/black/22.3.0/PatternGrammar3.8.10.final.0.pickle[0m | |
| [2023-10-31 00:04:39,364] INFO: [32mWriting failed: [Errno 2] No such file or directory: '/u/maximilian.kannen/.cache/black/22.3.0/tmp_ii5mp6k'[0m | |
| [2023-10-31 00:04:39,823] INFO: [32mRun time: 0:00:00 CPU: 0.00% RSS: 79MB VMS: 294MB[0m | |
| [2023-10-31 00:04:44,894] INFO: [32mStart Job: Job<alias/experiments/switchboard/ctc/feat/train_nn/conformer_bs5k_scf_baseline_big work/i6_core/returnn/training/ReturnnTrainingJob.wQIA3Rc1nLak> Task: run[0m | |
| [2023-10-31 00:04:44,895] INFO: [32mInputs:[0m | |
| [2023-10-31 00:04:44,895] INFO: [32m/u/vieting/setups/swb/20230406_feat/dependencies/allophones_blank[0m | |
| [2023-10-31 00:04:44,895] INFO: [32m/u/vieting/setups/swb/20230406_feat/dependencies/state-tying_blank[0m | |
| [2023-10-31 00:04:44,896] INFO: [32m/usr/bin/python3[0m | |
| [2023-10-31 00:04:44,896] INFO: [32m/work/asr4/vieting/programs/rasr/20230707/rasr/arch/linux-x86_64-standard/nn-trainer.linux-x86_64-standard[0m | |
| [2023-10-31 00:04:44,896] INFO: [32m/u/maximilian.kannen/setups/20230406_feat/work/i6_core/corpus/filter/FilterSegmentsByListJob.Fzh6DWEkIA5y/output/segments.1[0m | |
| [2023-10-31 00:04:44,896] INFO: [32m/u/maximilian.kannen/setups/20230406_feat/work/i6_core/corpus/filter/FilterSegmentsByListJob.SVlbt6fqP4Jn/output/segments.1[0m | |
| [2023-10-31 00:04:44,896] INFO: [32m/u/maximilian.kannen/setups/20230406_feat/work/i6_core/corpus/filter/FilterSegmentsByListJob.nrKcBIdsMBZm/output/segments.1[0m | |
| [2023-10-31 00:04:44,896] INFO: [32m/u/maximilian.kannen/setups/20230406_feat/work/i6_core/datasets/switchboard/CreateSwitchboardBlissCorpusJob.Z1EMi4TdrUS6/output/swb.corpus.xml.gz[0m | |
| [2023-10-31 00:04:44,896] INFO: [32m/u/maximilian.kannen/setups/20230406_feat/work/i6_core/returnn/oggzip/BlissToOggZipJob.lAFM8R9mzLpI/output/out.ogg.zip[0m | |
| [2023-10-31 00:04:44,897] INFO: [32m/u/maximilian.kannen/setups/20230406_feat/work/i6_core/text/processing/TailJob.RiSM6fe2XipO/output/out.gz[0m | |
| [2023-10-31 00:04:44,897] INFO: [32m/u/maximilian.kannen/setups/20230406_feat/work/i6_core/tools/git/CloneGitRepositoryJob.FigHMwYJhhef/output/repository[0m | |
| [2023-10-31 00:04:44,897] INFO: [32m/u/maximilian.kannen/setups/20230406_feat/work/i6_experiments/users/berger/recipe/lexicon/modification/MakeBlankLexiconJob.N8RlHYKzilei/output/lexicon.xml[0m | |
| Uname: uname_result(system='Linux', node='cn-505', release='5.15.0-39-generic', version='#42-Ubuntu SMP Thu Jun 9 23:42:32 UTC 2022', machine='x86_64', processor='x86_64') | |
| Load: (3.68, 3.74, 4.25) | |
| [2023-10-31 00:04:44,897] INFO: [32m------------------------------------------------------------[0m | |
| [2023-10-31 00:04:44,897] INFO: [32mStarting subtask for arg id: 0 args: [][0m | |
| [2023-10-31 00:04:44,897] INFO: [32m------------------------------------------------------------[0m | |
| RETURNN starting up, version 1.20231026.144554+git.d62891f6.dirty, date/time 2023-10-31-00-04-45 (UTC+0100), pid 1781720, cwd /work/asr3/vieting/hiwis/kannen/sisyphus_work_dirs/swb/i6_core/returnn/training/ReturnnTrainingJob.wQIA3Rc1nLak/work, Python /usr/bin/python3 | |
| RETURNN command line options: ['/u/maximilian.kannen/setups/20230406_feat/work/i6_core/returnn/training/ReturnnTrainingJob.wQIA3Rc1nLak/output/returnn.config'] | |
| Hostname: cn-505 | |
| MEMORY: main proc python3(1781720) initial: rss=43.6MB pss=33.8MB uss=24.0MB shared=19.6MB | |
| MEMORY: total (1 procs): pss=33.8MB uss=24.0MB | |
| TensorFlow: 2.8.0 (unknown) (<not-under-git> in /usr/local/lib/python3.8/dist-packages/tensorflow) | |
| Use num_threads=1 (but min 2) via OMP_NUM_THREADS. | |
| Setup TF inter and intra global thread pools, num_threads 2, session opts {'log_device_placement': False, 'device_count': {'GPU': 0}, 'intra_op_parallelism_threads': 2, 'inter_op_parallelism_threads': 2}. | |
| 2023-10-31 00:04:48.897402: I tensorflow/core/platform/cpu_feature_guard.cc:152] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE3 SSE4.1 SSE4.2 AVX | |
| To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. | |
| CUDA_VISIBLE_DEVICES is set to '0'. | |
| Collecting TensorFlow device list... | |
| [2023-10-31 00:04:49,910] INFO: [32mRun time: 0:00:10 CPU: 0.10% RSS: 699MB VMS: 8.41GB[0m | |
| 2023-10-31 00:04:50.839148: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /device:GPU:0 with 20396 MB memory: -> device: 0, name: NVIDIA A10, pci bus id: 0000:17:00.0, compute capability: 8.6 | |
| Local devices available to TensorFlow: | |
| 1/2: name: "/device:CPU:0" | |
| device_type: "CPU" | |
| memory_limit: 268435456 | |
| locality { | |
| } | |
| incarnation: 1509809284722411289 | |
| xla_global_id: -1 | |
| 2/2: name: "/device:GPU:0" | |
| device_type: "GPU" | |
| memory_limit: 21387673600 | |
| locality { | |
| bus_id: 1 | |
| links { | |
| } | |
| } | |
| incarnation: 5032034194876181478 | |
| physical_device_desc: "device: 0, name: NVIDIA A10, pci bus id: 0000:17:00.0, compute capability: 8.6" | |
| xla_global_id: 416903419 | |
| Using gpu device 0: NVIDIA A10 | |
| Hostname 'cn-505', GPU 0, GPU-dev-name 'NVIDIA A10', GPU-memory 19.9GB | |
| LOG: connected to ('10.6.100.1', 10321) | |
| LOG: destination: /var/tmp/maximilian.kannen/work/asr4/vieting/setups/swb/work/20230406_feat/i6_core/returnn/oggzip/BlissToOggZipJob.lAFM8R9mzLpI/output/out.ogg.zip | |
| LOG: using existing file | |
| MEMORY: main proc python3(1781720) increased RSS: rss=0.9GB pss=0.9GB uss=0.9GB shared=26.9MB | |
| MEMORY: total (1 procs): pss=0.9GB uss=0.9GB | |
| LOG: connected to ('10.6.100.1', 10321) | |
| LOG: destination: /var/tmp/maximilian.kannen/work/asr4/vieting/setups/swb/work/20230406_feat/i6_core/returnn/oggzip/BlissToOggZipJob.lAFM8R9mzLpI/output/out.ogg.zip | |
| LOG: using existing file | |
| LOG: connected to ('10.6.100.1', 10321) | |
| LOG: destination: /var/tmp/maximilian.kannen/work/asr4/vieting/setups/swb/work/20230406_feat/i6_core/returnn/oggzip/BlissToOggZipJob.lAFM8R9mzLpI/output/out.ogg.zip | |
| LOG: using existing file | |
| LOG: connected to ('10.6.100.1', 10321) | |
| LOG: destination: /var/tmp/maximilian.kannen/work/asr4/vieting/setups/swb/work/20230406_feat/i6_core/returnn/oggzip/BlissToOggZipJob.lAFM8R9mzLpI/output/out.ogg.zip | |
| LOG: using existing file | |
| Train data: | |
| input: 1 x 1 | |
| output: {'raw': {'dtype': 'string', 'shape': ()}, 'orth': [256, 1], 'data': [1, 2]} | |
| MultiProcDataset, sequences: 249229, frames: unknown | |
| Dev data: | |
| [2023-10-31 00:04:54,925] INFO: [32mRun time: 0:00:15 CPU: 0.40% RSS: 1.78GB VMS: 10.31GB[0m | |
| MEMORY: main proc python3(1781720) increased RSS: rss=1.3GB pss=1.2GB uss=1.2GB shared=28.7MB | |
| MEMORY: sub proc python3(1781780) initial: rss=10.7MB pss=6.0MB uss=5.0MB shared=5.7MB | |
| MEMORY: sub proc MPD seq order(1781781) initial: rss=571.1MB pss=554.4MB uss=549.8MB shared=21.3MB | |
| MEMORY: sub proc MPD worker(1781782) initial: rss=42.1MB pss=25.9MB uss=21.8MB shared=20.3MB | |
| MEMORY: sub proc MPD worker(1781783) initial: rss=42.1MB pss=25.9MB uss=21.9MB shared=20.2MB | |
| MEMORY: total (5 procs): pss=1.8GB uss=1.8GB | |
| OggZipDataset, sequences: 300, frames: unknown | |
| Learning-rate-control: loading file learning_rates | |
| Setup TF session with options {'log_device_placement': False, 'device_count': {'GPU': 1}} ... | |
| 2023-10-31 00:04:56.635529: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 20396 MB memory: -> device: 0, name: NVIDIA A10, pci bus id: 0000:17:00.0, compute capability: 8.6 | |
| layer /'data': [B,T|'time:var:extern_data:data'[B],F|F'feature:data'(1)] float32 | |
| layer /features/'conv_h_filter': ['conv_h_filter:static:0'(128),'conv_h_filter:static:1'(1),F|F'conv_h_filter:static:2'(150)] float32 | |
| layer /features/'conv_h': [B,T|'⌈((-63+time:var:extern_data:data)+-64)/5⌉'[B],F|F'conv_h:channel'(150)] float32 | |
| layer /features/'conv_h_act': [B,T|'⌈((-63+time:var:extern_data:data)+-64)/5⌉'[B],F|F'conv_h:channel'(150)] float32 | |
| layer /features/'conv_h_split': [B,T|'⌈((-63+time:var:extern_data:data)+-64)/5⌉'[B],F'conv_h:channel'(150),F|F'conv_h_split_split_dims1'(1)] float32 | |
| DEPRECATION WARNING: Explicitly specify in_spatial_dims when there is more than one spatial dim in the input. | |
| This will be disallowed with behavior_version 8. | |
| layer /features/'conv_l': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/16⌉'[B],F'conv_h:channel'(150),F|F'conv_l:channel'(5)] float32 | |
| layer /features/'conv_l_merge': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/16⌉'[B],F|F'conv_h:channel*conv_l:channel'(750)] float32 | |
| DEPRECATION WARNING: MergeDimsLayer, only keep_order=True is allowed | |
| This will be disallowed with behavior_version 6. | |
| layer /features/'conv_l_act_no_norm': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/16⌉'[B],F|F'conv_h:channel*conv_l:channel'(750)] float32 | |
| layer /features/'conv_l_act': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/16⌉'[B],F|F'conv_h:channel*conv_l:channel'(750)] float32 | |
| layer /features/'output': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/16⌉'[B],F|F'conv_h:channel*conv_l:channel'(750)] float32 | |
| layer /'features': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/16⌉'[B],F|F'conv_h:channel*conv_l:channel'(750)] float32 | |
| layer /'specaug': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/16⌉'[B],F|F'conv_h:channel*conv_l:channel'(750)] float32 | |
| WARNING:tensorflow:From /work/asr3/vieting/hiwis/kannen/sisyphus_work_dirs/swb/i6_core/tools/git/CloneGitRepositoryJob.FigHMwYJhhef/output/repository/returnn/tf/network.py:2462: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. | |
| Instructions for updating: | |
| Call initializer instance with the dtype argument instead of passing it to the constructor | |
| layer /'conv_source': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/16⌉'[B],F'conv_h:channel*conv_l:channel'(750),F|F'conv_source_split_dims1'(1)] float32 | |
| layer /'conv_1': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/16⌉'[B],F'conv_h:channel*conv_l:channel'(750),F|F'conv_1:channel'(32)] float32 | |
| layer /'conv_1_pool': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/16⌉'[B],'conv_h:channel*conv_l:channel//2'(375),F|F'conv_1:channel'(32)] float32 | |
| layer /'conv_2': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/32⌉'[B],'conv_h:channel*conv_l:channel//2'(375),F|F'conv_2:channel'(64)] float32 | |
| layer /'conv_3': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],'conv_h:channel*conv_l:channel//2'(375),F|F'conv_3:channel'(64)] float32 | |
| layer /'conv_merged': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'(conv_h:channel*conv_l:channel//2)*conv_3:channel'(24000)] float32 | |
| layer /'input_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'input_linear:feature-dense'(512)] float32 | |
| layer /'input_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'input_linear:feature-dense'(512)] float32 | |
| layer /'conformer_1_ffmod_1_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'input_linear:feature-dense'(512)] float32 | |
| layer /'conformer_1_ffmod_1_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_ffmod_1_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_1_ffmod_1_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_1_ffmod_1_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_1_ffmod_1_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_1_conv_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_1_conv_mod_pointwise_conv_1': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_conv_mod_pointwise_conv_1:feature-dense'(1024)] float32 | |
| layer /'conformer_1_conv_mod_glu': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'(conformer_1_conv_mod_pointwise_conv_1:feature-dense)//2'(512)] float32 | |
| layer /'conformer_1_conv_mod_depthwise_conv': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_1_conv_mod_bn': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_conv_mod_depthwise_conv:channel'(512)] float32 | |
| DEPRECATION WARNING: batch_norm masked_time should be specified explicitly | |
| This will be disallowed with behavior_version 12. | |
| WARNING:tensorflow:From /work/asr3/vieting/hiwis/kannen/sisyphus_work_dirs/swb/i6_core/tools/git/CloneGitRepositoryJob.FigHMwYJhhef/output/repository/returnn/tf/util/basic.py:1725: calling Ones.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. | |
| Instructions for updating: | |
| Call initializer instance with the dtype argument instead of passing it to the constructor | |
| layer /'conformer_1_conv_mod_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_1_conv_mod_pointwise_conv_2': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_1_conv_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_1_conv_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_1_mhsa_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_1_mhsa_mod_relpos_encoding': [T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_mhsa_mod_relpos_encoding_rel_pos_enc_feat'(64)] float32 | |
| layer /'conformer_1_mhsa_mod_self_attention': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_1_mhsa_mod_att_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_1_mhsa_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_1_mhsa_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_1_ffmod_2_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_1_ffmod_2_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_ffmod_2_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_1_ffmod_2_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_1_ffmod_2_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_1_ffmod_2_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_1_output': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_2_ffmod_1_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_1_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_2_ffmod_1_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_ffmod_1_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_2_ffmod_1_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_2_ffmod_1_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_2_ffmod_1_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_2_conv_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_2_conv_mod_pointwise_conv_1': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_conv_mod_pointwise_conv_1:feature-dense'(1024)] float32 | |
| layer /'conformer_2_conv_mod_glu': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'(conformer_2_conv_mod_pointwise_conv_1:feature-dense)//2'(512)] float32 | |
| layer /'conformer_2_conv_mod_depthwise_conv': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_2_conv_mod_bn': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_2_conv_mod_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_2_conv_mod_pointwise_conv_2': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_2_conv_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_2_conv_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_2_mhsa_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_2_mhsa_mod_relpos_encoding': [T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_mhsa_mod_relpos_encoding_rel_pos_enc_feat'(64)] float32 | |
| layer /'conformer_2_mhsa_mod_self_attention': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_2_mhsa_mod_att_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_2_mhsa_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_2_mhsa_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_2_ffmod_2_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_2_ffmod_2_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_ffmod_2_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_2_ffmod_2_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_2_ffmod_2_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_2_ffmod_2_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_2_output': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_3_ffmod_1_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_2_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_3_ffmod_1_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_ffmod_1_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_3_ffmod_1_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_3_ffmod_1_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_3_ffmod_1_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_3_conv_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_3_conv_mod_pointwise_conv_1': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_conv_mod_pointwise_conv_1:feature-dense'(1024)] float32 | |
| layer /'conformer_3_conv_mod_glu': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'(conformer_3_conv_mod_pointwise_conv_1:feature-dense)//2'(512)] float32 | |
| layer /'conformer_3_conv_mod_depthwise_conv': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_3_conv_mod_bn': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_3_conv_mod_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_3_conv_mod_pointwise_conv_2': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_3_conv_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_3_conv_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_3_mhsa_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_3_mhsa_mod_relpos_encoding': [T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_mhsa_mod_relpos_encoding_rel_pos_enc_feat'(64)] float32 | |
| layer /'conformer_3_mhsa_mod_self_attention': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_3_mhsa_mod_att_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_3_mhsa_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_3_mhsa_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_3_ffmod_2_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_3_ffmod_2_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_ffmod_2_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_3_ffmod_2_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_3_ffmod_2_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_3_ffmod_2_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_3_output': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_4_ffmod_1_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_3_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_4_ffmod_1_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_ffmod_1_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_4_ffmod_1_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_4_ffmod_1_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_4_ffmod_1_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_4_conv_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_4_conv_mod_pointwise_conv_1': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_conv_mod_pointwise_conv_1:feature-dense'(1024)] float32 | |
| layer /'conformer_4_conv_mod_glu': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'(conformer_4_conv_mod_pointwise_conv_1:feature-dense)//2'(512)] float32 | |
| layer /'conformer_4_conv_mod_depthwise_conv': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_4_conv_mod_bn': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_4_conv_mod_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_4_conv_mod_pointwise_conv_2': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_4_conv_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_4_conv_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_4_mhsa_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_4_mhsa_mod_relpos_encoding': [T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_mhsa_mod_relpos_encoding_rel_pos_enc_feat'(64)] float32 | |
| layer /'conformer_4_mhsa_mod_self_attention': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_4_mhsa_mod_att_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_4_mhsa_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_4_mhsa_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_4_ffmod_2_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_4_ffmod_2_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_ffmod_2_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_4_ffmod_2_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_4_ffmod_2_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_4_ffmod_2_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_4_output': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_5_ffmod_1_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_4_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_5_ffmod_1_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_ffmod_1_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_5_ffmod_1_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_5_ffmod_1_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_5_ffmod_1_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_5_conv_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_5_conv_mod_pointwise_conv_1': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_conv_mod_pointwise_conv_1:feature-dense'(1024)] float32 | |
| layer /'conformer_5_conv_mod_glu': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'(conformer_5_conv_mod_pointwise_conv_1:feature-dense)//2'(512)] float32 | |
| layer /'conformer_5_conv_mod_depthwise_conv': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_5_conv_mod_bn': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_5_conv_mod_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_5_conv_mod_pointwise_conv_2': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_5_conv_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_5_conv_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_5_mhsa_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_5_mhsa_mod_relpos_encoding': [T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_mhsa_mod_relpos_encoding_rel_pos_enc_feat'(64)] float32 | |
| layer /'conformer_5_mhsa_mod_self_attention': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_5_mhsa_mod_att_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_5_mhsa_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_5_mhsa_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_5_ffmod_2_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_5_ffmod_2_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_ffmod_2_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_5_ffmod_2_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_5_ffmod_2_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_5_ffmod_2_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_5_output': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_6_ffmod_1_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_5_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_6_ffmod_1_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_ffmod_1_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_6_ffmod_1_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_6_ffmod_1_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_6_ffmod_1_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_6_conv_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_6_conv_mod_pointwise_conv_1': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_conv_mod_pointwise_conv_1:feature-dense'(1024)] float32 | |
| layer /'conformer_6_conv_mod_glu': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'(conformer_6_conv_mod_pointwise_conv_1:feature-dense)//2'(512)] float32 | |
| layer /'conformer_6_conv_mod_depthwise_conv': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_6_conv_mod_bn': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_6_conv_mod_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_6_conv_mod_pointwise_conv_2': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_6_conv_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_6_conv_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_6_mhsa_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_6_mhsa_mod_relpos_encoding': [T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_mhsa_mod_relpos_encoding_rel_pos_enc_feat'(64)] float32 | |
| layer /'conformer_6_mhsa_mod_self_attention': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_6_mhsa_mod_att_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_6_mhsa_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_6_mhsa_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_6_ffmod_2_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_6_ffmod_2_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_ffmod_2_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_6_ffmod_2_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_6_ffmod_2_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_6_ffmod_2_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_6_output': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_7_ffmod_1_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_6_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_7_ffmod_1_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_ffmod_1_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_7_ffmod_1_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_7_ffmod_1_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_7_ffmod_1_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_7_conv_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_7_conv_mod_pointwise_conv_1': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_conv_mod_pointwise_conv_1:feature-dense'(1024)] float32 | |
| layer /'conformer_7_conv_mod_glu': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'(conformer_7_conv_mod_pointwise_conv_1:feature-dense)//2'(512)] float32 | |
| layer /'conformer_7_conv_mod_depthwise_conv': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_7_conv_mod_bn': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_7_conv_mod_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_7_conv_mod_pointwise_conv_2': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_7_conv_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_7_conv_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_7_mhsa_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_7_mhsa_mod_relpos_encoding': [T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_mhsa_mod_relpos_encoding_rel_pos_enc_feat'(64)] float32 | |
| layer /'conformer_7_mhsa_mod_self_attention': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_7_mhsa_mod_att_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_7_mhsa_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_7_mhsa_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_7_ffmod_2_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_7_ffmod_2_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_ffmod_2_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_7_ffmod_2_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_7_ffmod_2_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_7_ffmod_2_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_7_output': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_8_ffmod_1_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_7_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_8_ffmod_1_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_ffmod_1_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_8_ffmod_1_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_8_ffmod_1_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_8_ffmod_1_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_8_conv_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_8_conv_mod_pointwise_conv_1': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_conv_mod_pointwise_conv_1:feature-dense'(1024)] float32 | |
| layer /'conformer_8_conv_mod_glu': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'(conformer_8_conv_mod_pointwise_conv_1:feature-dense)//2'(512)] float32 | |
| layer /'conformer_8_conv_mod_depthwise_conv': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_8_conv_mod_bn': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_8_conv_mod_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_8_conv_mod_pointwise_conv_2': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_8_conv_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_8_conv_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_8_mhsa_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_8_mhsa_mod_relpos_encoding': [T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_mhsa_mod_relpos_encoding_rel_pos_enc_feat'(64)] float32 | |
| layer /'conformer_8_mhsa_mod_self_attention': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| [2023-10-31 00:04:59,941] INFO: [32mRun time: 0:00:20 CPU: 0.40% RSS: 2.10GB VMS: 10.66GB[0m | |
| layer /'conformer_8_mhsa_mod_att_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_8_mhsa_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_8_mhsa_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_8_ffmod_2_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_8_ffmod_2_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_ffmod_2_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_8_ffmod_2_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_8_ffmod_2_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_8_ffmod_2_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_8_output': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_9_ffmod_1_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_8_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_9_ffmod_1_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_ffmod_1_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_9_ffmod_1_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_9_ffmod_1_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_9_ffmod_1_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_9_conv_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_9_conv_mod_pointwise_conv_1': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_conv_mod_pointwise_conv_1:feature-dense'(1024)] float32 | |
| layer /'conformer_9_conv_mod_glu': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'(conformer_9_conv_mod_pointwise_conv_1:feature-dense)//2'(512)] float32 | |
| layer /'conformer_9_conv_mod_depthwise_conv': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_9_conv_mod_bn': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_9_conv_mod_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_9_conv_mod_pointwise_conv_2': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_9_conv_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_9_conv_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_9_mhsa_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_9_mhsa_mod_relpos_encoding': [T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_mhsa_mod_relpos_encoding_rel_pos_enc_feat'(64)] float32 | |
| layer /'conformer_9_mhsa_mod_self_attention': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_9_mhsa_mod_att_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_9_mhsa_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_9_mhsa_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_9_ffmod_2_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_9_ffmod_2_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_ffmod_2_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_9_ffmod_2_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_9_ffmod_2_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_9_ffmod_2_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_9_output': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_10_ffmod_1_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_9_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_10_ffmod_1_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_ffmod_1_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_10_ffmod_1_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_10_ffmod_1_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_10_ffmod_1_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_10_conv_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_10_conv_mod_pointwise_conv_1': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_conv_mod_pointwise_conv_1:feature-dense'(1024)] float32 | |
| layer /'conformer_10_conv_mod_glu': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'(conformer_10_conv_mod_pointwise_conv_1:feature-dense)//2'(512)] float32 | |
| layer /'conformer_10_conv_mod_depthwise_conv': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_10_conv_mod_bn': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_10_conv_mod_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_10_conv_mod_pointwise_conv_2': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_10_conv_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_10_conv_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_10_mhsa_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_10_mhsa_mod_relpos_encoding': [T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_mhsa_mod_relpos_encoding_rel_pos_enc_feat'(64)] float32 | |
| layer /'conformer_10_mhsa_mod_self_attention': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_10_mhsa_mod_att_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_10_mhsa_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_10_mhsa_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_10_ffmod_2_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_10_ffmod_2_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_ffmod_2_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_10_ffmod_2_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_10_ffmod_2_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_10_ffmod_2_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_10_output': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_11_ffmod_1_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_10_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_11_ffmod_1_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_ffmod_1_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_11_ffmod_1_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_11_ffmod_1_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_11_ffmod_1_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_11_conv_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_11_conv_mod_pointwise_conv_1': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_conv_mod_pointwise_conv_1:feature-dense'(1024)] float32 | |
| layer /'conformer_11_conv_mod_glu': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'(conformer_11_conv_mod_pointwise_conv_1:feature-dense)//2'(512)] float32 | |
| layer /'conformer_11_conv_mod_depthwise_conv': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_11_conv_mod_bn': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_11_conv_mod_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_11_conv_mod_pointwise_conv_2': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_11_conv_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_11_conv_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_11_mhsa_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_11_mhsa_mod_relpos_encoding': [T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_mhsa_mod_relpos_encoding_rel_pos_enc_feat'(64)] float32 | |
| layer /'conformer_11_mhsa_mod_self_attention': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_11_mhsa_mod_att_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_11_mhsa_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_11_mhsa_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_11_ffmod_2_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_11_ffmod_2_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_ffmod_2_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_11_ffmod_2_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_11_ffmod_2_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_11_ffmod_2_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_11_output': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_12_ffmod_1_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_11_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_12_ffmod_1_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_ffmod_1_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_12_ffmod_1_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_12_ffmod_1_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_12_ffmod_1_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_12_conv_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_ffmod_1_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_12_conv_mod_pointwise_conv_1': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_conv_mod_pointwise_conv_1:feature-dense'(1024)] float32 | |
| layer /'conformer_12_conv_mod_glu': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'(conformer_12_conv_mod_pointwise_conv_1:feature-dense)//2'(512)] float32 | |
| layer /'conformer_12_conv_mod_depthwise_conv': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_12_conv_mod_bn': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_conv_mod_depthwise_conv:channel'(512)] float32 | |
| MEMORY: main proc python3(1781720) increased RSS: rss=1.4GB pss=1.4GB uss=1.4GB shared=28.8MB | |
| MEMORY: total (5 procs): pss=2.0GB uss=2.0GB | |
| layer /'conformer_12_conv_mod_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_12_conv_mod_pointwise_conv_2': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_12_conv_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_12_conv_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_12_mhsa_mod_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_conv_mod_depthwise_conv:channel'(512)] float32 | |
| layer /'conformer_12_mhsa_mod_relpos_encoding': [T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_mhsa_mod_relpos_encoding_rel_pos_enc_feat'(64)] float32 | |
| layer /'conformer_12_mhsa_mod_self_attention': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_12_mhsa_mod_att_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_12_mhsa_mod_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_12_mhsa_mod_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_12_ffmod_2_ln': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_mhsa_mod_self_attention_self_att_feat'(512)] float32 | |
| layer /'conformer_12_ffmod_2_linear_swish': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_ffmod_2_linear_swish:feature-dense'(2048)] float32 | |
| layer /'conformer_12_ffmod_2_dropout_linear': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_12_ffmod_2_dropout': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_12_ffmod_2_half_res_add': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'conformer_12_output': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| layer /'encoder': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'conformer_12_ffmod_2_dropout_linear:feature-dense'(512)] float32 | |
| 2023-10-31 00:05:01.756689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 20396 MB memory: -> device: 0, name: NVIDIA A10, pci bus id: 0000:17:00.0, compute capability: 8.6 | |
| layer /'output': [B,T|'⌈((-19+(⌈((-63+time:var:extern_data:data)+-64)/5⌉))+-20)/64⌉'[B],F|F'output:feature-dense'(88)] float32 | |
| WARNING:tensorflow:From /work/asr3/vieting/hiwis/kannen/sisyphus_work_dirs/swb/i6_core/tools/git/CloneGitRepositoryJob.FigHMwYJhhef/output/repository/returnn/tf/sprint.py:54: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version. | |
| Instructions for updating: | |
| tf.py_func is deprecated in TF V2. Instead, there are two | |
| options available in V2. | |
| - tf.py_function takes a python function which manipulates tf eager | |
| tensors instead of numpy arrays. It's easy to convert a tf eager tensor to | |
| an ndarray (just call tensor.numpy()) but having access to eager tensors | |
| means `tf.py_function`s can use accelerators such as GPUs as well as | |
| being differentiable using a gradient tape. | |
| - tf.numpy_function maintains the semantics of the deprecated tf.py_func | |
| (it is not differentiable, and manipulates numpy arrays). It drops the | |
| stateful argument making all functions stateful. | |
| Network layer topology: | |
| extern data: data: Tensor{[B,T|'time:var:extern_data:data'[B],F|F'feature:data'(1)]}, seq_tag: Tensor{[B?], dtype='string'} | |
| used data keys: ['data', 'seq_tag'] | |
| layers: | |
| layer batch_norm 'conformer_10_conv_mod_bn' #: 512 | |
| layer conv 'conformer_10_conv_mod_depthwise_conv' #: 512 | |
| layer copy 'conformer_10_conv_mod_dropout' #: 512 | |
| layer gating 'conformer_10_conv_mod_glu' #: 512 | |
| layer layer_norm 'conformer_10_conv_mod_ln' #: 512 | |
| layer linear 'conformer_10_conv_mod_pointwise_conv_1' #: 1024 | |
| layer linear 'conformer_10_conv_mod_pointwise_conv_2' #: 512 | |
| layer combine 'conformer_10_conv_mod_res_add' #: 512 | |
| layer activation 'conformer_10_conv_mod_swish' #: 512 | |
| layer copy 'conformer_10_ffmod_1_dropout' #: 512 | |
| layer linear 'conformer_10_ffmod_1_dropout_linear' #: 512 | |
| layer eval 'conformer_10_ffmod_1_half_res_add' #: 512 | |
| layer linear 'conformer_10_ffmod_1_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_10_ffmod_1_ln' #: 512 | |
| layer copy 'conformer_10_ffmod_2_dropout' #: 512 | |
| layer linear 'conformer_10_ffmod_2_dropout_linear' #: 512 | |
| layer eval 'conformer_10_ffmod_2_half_res_add' #: 512 | |
| layer linear 'conformer_10_ffmod_2_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_10_ffmod_2_ln' #: 512 | |
| layer linear 'conformer_10_mhsa_mod_att_linear' #: 512 | |
| layer copy 'conformer_10_mhsa_mod_dropout' #: 512 | |
| layer layer_norm 'conformer_10_mhsa_mod_ln' #: 512 | |
| layer relative_positional_encoding 'conformer_10_mhsa_mod_relpos_encoding' #: 64 | |
| layer combine 'conformer_10_mhsa_mod_res_add' #: 512 | |
| layer self_attention 'conformer_10_mhsa_mod_self_attention' #: 512 | |
| layer layer_norm 'conformer_10_output' #: 512 | |
| layer batch_norm 'conformer_11_conv_mod_bn' #: 512 | |
| layer conv 'conformer_11_conv_mod_depthwise_conv' #: 512 | |
| layer copy 'conformer_11_conv_mod_dropout' #: 512 | |
| layer gating 'conformer_11_conv_mod_glu' #: 512 | |
| layer layer_norm 'conformer_11_conv_mod_ln' #: 512 | |
| layer linear 'conformer_11_conv_mod_pointwise_conv_1' #: 1024 | |
| layer linear 'conformer_11_conv_mod_pointwise_conv_2' #: 512 | |
| layer combine 'conformer_11_conv_mod_res_add' #: 512 | |
| layer activation 'conformer_11_conv_mod_swish' #: 512 | |
| layer copy 'conformer_11_ffmod_1_dropout' #: 512 | |
| layer linear 'conformer_11_ffmod_1_dropout_linear' #: 512 | |
| layer eval 'conformer_11_ffmod_1_half_res_add' #: 512 | |
| layer linear 'conformer_11_ffmod_1_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_11_ffmod_1_ln' #: 512 | |
| layer copy 'conformer_11_ffmod_2_dropout' #: 512 | |
| layer linear 'conformer_11_ffmod_2_dropout_linear' #: 512 | |
| layer eval 'conformer_11_ffmod_2_half_res_add' #: 512 | |
| layer linear 'conformer_11_ffmod_2_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_11_ffmod_2_ln' #: 512 | |
| layer linear 'conformer_11_mhsa_mod_att_linear' #: 512 | |
| layer copy 'conformer_11_mhsa_mod_dropout' #: 512 | |
| layer layer_norm 'conformer_11_mhsa_mod_ln' #: 512 | |
| layer relative_positional_encoding 'conformer_11_mhsa_mod_relpos_encoding' #: 64 | |
| layer combine 'conformer_11_mhsa_mod_res_add' #: 512 | |
| layer self_attention 'conformer_11_mhsa_mod_self_attention' #: 512 | |
| layer layer_norm 'conformer_11_output' #: 512 | |
| layer batch_norm 'conformer_12_conv_mod_bn' #: 512 | |
| layer conv 'conformer_12_conv_mod_depthwise_conv' #: 512 | |
| layer copy 'conformer_12_conv_mod_dropout' #: 512 | |
| layer gating 'conformer_12_conv_mod_glu' #: 512 | |
| layer layer_norm 'conformer_12_conv_mod_ln' #: 512 | |
| layer linear 'conformer_12_conv_mod_pointwise_conv_1' #: 1024 | |
| layer linear 'conformer_12_conv_mod_pointwise_conv_2' #: 512 | |
| layer combine 'conformer_12_conv_mod_res_add' #: 512 | |
| layer activation 'conformer_12_conv_mod_swish' #: 512 | |
| layer copy 'conformer_12_ffmod_1_dropout' #: 512 | |
| layer linear 'conformer_12_ffmod_1_dropout_linear' #: 512 | |
| layer eval 'conformer_12_ffmod_1_half_res_add' #: 512 | |
| layer linear 'conformer_12_ffmod_1_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_12_ffmod_1_ln' #: 512 | |
| layer copy 'conformer_12_ffmod_2_dropout' #: 512 | |
| layer linear 'conformer_12_ffmod_2_dropout_linear' #: 512 | |
| layer eval 'conformer_12_ffmod_2_half_res_add' #: 512 | |
| layer linear 'conformer_12_ffmod_2_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_12_ffmod_2_ln' #: 512 | |
| layer linear 'conformer_12_mhsa_mod_att_linear' #: 512 | |
| layer copy 'conformer_12_mhsa_mod_dropout' #: 512 | |
| layer layer_norm 'conformer_12_mhsa_mod_ln' #: 512 | |
| layer relative_positional_encoding 'conformer_12_mhsa_mod_relpos_encoding' #: 64 | |
| layer combine 'conformer_12_mhsa_mod_res_add' #: 512 | |
| layer self_attention 'conformer_12_mhsa_mod_self_attention' #: 512 | |
| layer layer_norm 'conformer_12_output' #: 512 | |
| layer batch_norm 'conformer_1_conv_mod_bn' #: 512 | |
| layer conv 'conformer_1_conv_mod_depthwise_conv' #: 512 | |
| layer copy 'conformer_1_conv_mod_dropout' #: 512 | |
| layer gating 'conformer_1_conv_mod_glu' #: 512 | |
| layer layer_norm 'conformer_1_conv_mod_ln' #: 512 | |
| layer linear 'conformer_1_conv_mod_pointwise_conv_1' #: 1024 | |
| layer linear 'conformer_1_conv_mod_pointwise_conv_2' #: 512 | |
| layer combine 'conformer_1_conv_mod_res_add' #: 512 | |
| layer activation 'conformer_1_conv_mod_swish' #: 512 | |
| layer copy 'conformer_1_ffmod_1_dropout' #: 512 | |
| layer linear 'conformer_1_ffmod_1_dropout_linear' #: 512 | |
| layer eval 'conformer_1_ffmod_1_half_res_add' #: 512 | |
| layer linear 'conformer_1_ffmod_1_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_1_ffmod_1_ln' #: 512 | |
| layer copy 'conformer_1_ffmod_2_dropout' #: 512 | |
| layer linear 'conformer_1_ffmod_2_dropout_linear' #: 512 | |
| layer eval 'conformer_1_ffmod_2_half_res_add' #: 512 | |
| layer linear 'conformer_1_ffmod_2_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_1_ffmod_2_ln' #: 512 | |
| layer linear 'conformer_1_mhsa_mod_att_linear' #: 512 | |
| layer copy 'conformer_1_mhsa_mod_dropout' #: 512 | |
| layer layer_norm 'conformer_1_mhsa_mod_ln' #: 512 | |
| layer relative_positional_encoding 'conformer_1_mhsa_mod_relpos_encoding' #: 64 | |
| layer combine 'conformer_1_mhsa_mod_res_add' #: 512 | |
| layer self_attention 'conformer_1_mhsa_mod_self_attention' #: 512 | |
| layer layer_norm 'conformer_1_output' #: 512 | |
| layer batch_norm 'conformer_2_conv_mod_bn' #: 512 | |
| layer conv 'conformer_2_conv_mod_depthwise_conv' #: 512 | |
| layer copy 'conformer_2_conv_mod_dropout' #: 512 | |
| layer gating 'conformer_2_conv_mod_glu' #: 512 | |
| layer layer_norm 'conformer_2_conv_mod_ln' #: 512 | |
| layer linear 'conformer_2_conv_mod_pointwise_conv_1' #: 1024 | |
| layer linear 'conformer_2_conv_mod_pointwise_conv_2' #: 512 | |
| layer combine 'conformer_2_conv_mod_res_add' #: 512 | |
| layer activation 'conformer_2_conv_mod_swish' #: 512 | |
| layer copy 'conformer_2_ffmod_1_dropout' #: 512 | |
| layer linear 'conformer_2_ffmod_1_dropout_linear' #: 512 | |
| layer eval 'conformer_2_ffmod_1_half_res_add' #: 512 | |
| layer linear 'conformer_2_ffmod_1_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_2_ffmod_1_ln' #: 512 | |
| layer copy 'conformer_2_ffmod_2_dropout' #: 512 | |
| layer linear 'conformer_2_ffmod_2_dropout_linear' #: 512 | |
| layer eval 'conformer_2_ffmod_2_half_res_add' #: 512 | |
| layer linear 'conformer_2_ffmod_2_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_2_ffmod_2_ln' #: 512 | |
| layer linear 'conformer_2_mhsa_mod_att_linear' #: 512 | |
| layer copy 'conformer_2_mhsa_mod_dropout' #: 512 | |
| layer layer_norm 'conformer_2_mhsa_mod_ln' #: 512 | |
| layer relative_positional_encoding 'conformer_2_mhsa_mod_relpos_encoding' #: 64 | |
| layer combine 'conformer_2_mhsa_mod_res_add' #: 512 | |
| layer self_attention 'conformer_2_mhsa_mod_self_attention' #: 512 | |
| layer layer_norm 'conformer_2_output' #: 512 | |
| layer batch_norm 'conformer_3_conv_mod_bn' #: 512 | |
| layer conv 'conformer_3_conv_mod_depthwise_conv' #: 512 | |
| layer copy 'conformer_3_conv_mod_dropout' #: 512 | |
| layer gating 'conformer_3_conv_mod_glu' #: 512 | |
| layer layer_norm 'conformer_3_conv_mod_ln' #: 512 | |
| layer linear 'conformer_3_conv_mod_pointwise_conv_1' #: 1024 | |
| layer linear 'conformer_3_conv_mod_pointwise_conv_2' #: 512 | |
| layer combine 'conformer_3_conv_mod_res_add' #: 512 | |
| layer activation 'conformer_3_conv_mod_swish' #: 512 | |
| layer copy 'conformer_3_ffmod_1_dropout' #: 512 | |
| layer linear 'conformer_3_ffmod_1_dropout_linear' #: 512 | |
| layer eval 'conformer_3_ffmod_1_half_res_add' #: 512 | |
| layer linear 'conformer_3_ffmod_1_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_3_ffmod_1_ln' #: 512 | |
| layer copy 'conformer_3_ffmod_2_dropout' #: 512 | |
| layer linear 'conformer_3_ffmod_2_dropout_linear' #: 512 | |
| layer eval 'conformer_3_ffmod_2_half_res_add' #: 512 | |
| layer linear 'conformer_3_ffmod_2_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_3_ffmod_2_ln' #: 512 | |
| layer linear 'conformer_3_mhsa_mod_att_linear' #: 512 | |
| layer copy 'conformer_3_mhsa_mod_dropout' #: 512 | |
| layer layer_norm 'conformer_3_mhsa_mod_ln' #: 512 | |
| layer relative_positional_encoding 'conformer_3_mhsa_mod_relpos_encoding' #: 64 | |
| layer combine 'conformer_3_mhsa_mod_res_add' #: 512 | |
| layer self_attention 'conformer_3_mhsa_mod_self_attention' #: 512 | |
| layer layer_norm 'conformer_3_output' #: 512 | |
| layer batch_norm 'conformer_4_conv_mod_bn' #: 512 | |
| layer conv 'conformer_4_conv_mod_depthwise_conv' #: 512 | |
| layer copy 'conformer_4_conv_mod_dropout' #: 512 | |
| layer gating 'conformer_4_conv_mod_glu' #: 512 | |
| layer layer_norm 'conformer_4_conv_mod_ln' #: 512 | |
| layer linear 'conformer_4_conv_mod_pointwise_conv_1' #: 1024 | |
| layer linear 'conformer_4_conv_mod_pointwise_conv_2' #: 512 | |
| layer combine 'conformer_4_conv_mod_res_add' #: 512 | |
| layer activation 'conformer_4_conv_mod_swish' #: 512 | |
| layer copy 'conformer_4_ffmod_1_dropout' #: 512 | |
| layer linear 'conformer_4_ffmod_1_dropout_linear' #: 512 | |
| layer eval 'conformer_4_ffmod_1_half_res_add' #: 512 | |
| layer linear 'conformer_4_ffmod_1_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_4_ffmod_1_ln' #: 512 | |
| layer copy 'conformer_4_ffmod_2_dropout' #: 512 | |
| layer linear 'conformer_4_ffmod_2_dropout_linear' #: 512 | |
| layer eval 'conformer_4_ffmod_2_half_res_add' #: 512 | |
| layer linear 'conformer_4_ffmod_2_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_4_ffmod_2_ln' #: 512 | |
| layer linear 'conformer_4_mhsa_mod_att_linear' #: 512 | |
| layer copy 'conformer_4_mhsa_mod_dropout' #: 512 | |
| layer layer_norm 'conformer_4_mhsa_mod_ln' #: 512 | |
| layer relative_positional_encoding 'conformer_4_mhsa_mod_relpos_encoding' #: 64 | |
| layer combine 'conformer_4_mhsa_mod_res_add' #: 512 | |
| layer self_attention 'conformer_4_mhsa_mod_self_attention' #: 512 | |
| layer layer_norm 'conformer_4_output' #: 512 | |
| layer batch_norm 'conformer_5_conv_mod_bn' #: 512 | |
| layer conv 'conformer_5_conv_mod_depthwise_conv' #: 512 | |
| layer copy 'conformer_5_conv_mod_dropout' #: 512 | |
| layer gating 'conformer_5_conv_mod_glu' #: 512 | |
| layer layer_norm 'conformer_5_conv_mod_ln' #: 512 | |
| layer linear 'conformer_5_conv_mod_pointwise_conv_1' #: 1024 | |
| layer linear 'conformer_5_conv_mod_pointwise_conv_2' #: 512 | |
| layer combine 'conformer_5_conv_mod_res_add' #: 512 | |
| layer activation 'conformer_5_conv_mod_swish' #: 512 | |
| layer copy 'conformer_5_ffmod_1_dropout' #: 512 | |
| layer linear 'conformer_5_ffmod_1_dropout_linear' #: 512 | |
| layer eval 'conformer_5_ffmod_1_half_res_add' #: 512 | |
| layer linear 'conformer_5_ffmod_1_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_5_ffmod_1_ln' #: 512 | |
| layer copy 'conformer_5_ffmod_2_dropout' #: 512 | |
| layer linear 'conformer_5_ffmod_2_dropout_linear' #: 512 | |
| layer eval 'conformer_5_ffmod_2_half_res_add' #: 512 | |
| layer linear 'conformer_5_ffmod_2_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_5_ffmod_2_ln' #: 512 | |
| layer linear 'conformer_5_mhsa_mod_att_linear' #: 512 | |
| layer copy 'conformer_5_mhsa_mod_dropout' #: 512 | |
| layer layer_norm 'conformer_5_mhsa_mod_ln' #: 512 | |
| layer relative_positional_encoding 'conformer_5_mhsa_mod_relpos_encoding' #: 64 | |
| layer combine 'conformer_5_mhsa_mod_res_add' #: 512 | |
| layer self_attention 'conformer_5_mhsa_mod_self_attention' #: 512 | |
| layer layer_norm 'conformer_5_output' #: 512 | |
| layer batch_norm 'conformer_6_conv_mod_bn' #: 512 | |
| layer conv 'conformer_6_conv_mod_depthwise_conv' #: 512 | |
| layer copy 'conformer_6_conv_mod_dropout' #: 512 | |
| layer gating 'conformer_6_conv_mod_glu' #: 512 | |
| layer layer_norm 'conformer_6_conv_mod_ln' #: 512 | |
| layer linear 'conformer_6_conv_mod_pointwise_conv_1' #: 1024 | |
| layer linear 'conformer_6_conv_mod_pointwise_conv_2' #: 512 | |
| layer combine 'conformer_6_conv_mod_res_add' #: 512 | |
| layer activation 'conformer_6_conv_mod_swish' #: 512 | |
| layer copy 'conformer_6_ffmod_1_dropout' #: 512 | |
| layer linear 'conformer_6_ffmod_1_dropout_linear' #: 512 | |
| layer eval 'conformer_6_ffmod_1_half_res_add' #: 512 | |
| layer linear 'conformer_6_ffmod_1_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_6_ffmod_1_ln' #: 512 | |
| layer copy 'conformer_6_ffmod_2_dropout' #: 512 | |
| layer linear 'conformer_6_ffmod_2_dropout_linear' #: 512 | |
| layer eval 'conformer_6_ffmod_2_half_res_add' #: 512 | |
| layer linear 'conformer_6_ffmod_2_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_6_ffmod_2_ln' #: 512 | |
| layer linear 'conformer_6_mhsa_mod_att_linear' #: 512 | |
| layer copy 'conformer_6_mhsa_mod_dropout' #: 512 | |
| layer layer_norm 'conformer_6_mhsa_mod_ln' #: 512 | |
| layer relative_positional_encoding 'conformer_6_mhsa_mod_relpos_encoding' #: 64 | |
| layer combine 'conformer_6_mhsa_mod_res_add' #: 512 | |
| layer self_attention 'conformer_6_mhsa_mod_self_attention' #: 512 | |
| layer layer_norm 'conformer_6_output' #: 512 | |
| layer batch_norm 'conformer_7_conv_mod_bn' #: 512 | |
| layer conv 'conformer_7_conv_mod_depthwise_conv' #: 512 | |
| layer copy 'conformer_7_conv_mod_dropout' #: 512 | |
| layer gating 'conformer_7_conv_mod_glu' #: 512 | |
| layer layer_norm 'conformer_7_conv_mod_ln' #: 512 | |
| layer linear 'conformer_7_conv_mod_pointwise_conv_1' #: 1024 | |
| layer linear 'conformer_7_conv_mod_pointwise_conv_2' #: 512 | |
| layer combine 'conformer_7_conv_mod_res_add' #: 512 | |
| layer activation 'conformer_7_conv_mod_swish' #: 512 | |
| layer copy 'conformer_7_ffmod_1_dropout' #: 512 | |
| layer linear 'conformer_7_ffmod_1_dropout_linear' #: 512 | |
| layer eval 'conformer_7_ffmod_1_half_res_add' #: 512 | |
| layer linear 'conformer_7_ffmod_1_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_7_ffmod_1_ln' #: 512 | |
| layer copy 'conformer_7_ffmod_2_dropout' #: 512 | |
| layer linear 'conformer_7_ffmod_2_dropout_linear' #: 512 | |
| layer eval 'conformer_7_ffmod_2_half_res_add' #: 512 | |
| layer linear 'conformer_7_ffmod_2_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_7_ffmod_2_ln' #: 512 | |
| layer linear 'conformer_7_mhsa_mod_att_linear' #: 512 | |
| layer copy 'conformer_7_mhsa_mod_dropout' #: 512 | |
| layer layer_norm 'conformer_7_mhsa_mod_ln' #: 512 | |
| layer relative_positional_encoding 'conformer_7_mhsa_mod_relpos_encoding' #: 64 | |
| layer combine 'conformer_7_mhsa_mod_res_add' #: 512 | |
| layer self_attention 'conformer_7_mhsa_mod_self_attention' #: 512 | |
| layer layer_norm 'conformer_7_output' #: 512 | |
| layer batch_norm 'conformer_8_conv_mod_bn' #: 512 | |
| layer conv 'conformer_8_conv_mod_depthwise_conv' #: 512 | |
| layer copy 'conformer_8_conv_mod_dropout' #: 512 | |
| layer gating 'conformer_8_conv_mod_glu' #: 512 | |
| layer layer_norm 'conformer_8_conv_mod_ln' #: 512 | |
| layer linear 'conformer_8_conv_mod_pointwise_conv_1' #: 1024 | |
| layer linear 'conformer_8_conv_mod_pointwise_conv_2' #: 512 | |
| layer combine 'conformer_8_conv_mod_res_add' #: 512 | |
| layer activation 'conformer_8_conv_mod_swish' #: 512 | |
| layer copy 'conformer_8_ffmod_1_dropout' #: 512 | |
| layer linear 'conformer_8_ffmod_1_dropout_linear' #: 512 | |
| layer eval 'conformer_8_ffmod_1_half_res_add' #: 512 | |
| layer linear 'conformer_8_ffmod_1_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_8_ffmod_1_ln' #: 512 | |
| layer copy 'conformer_8_ffmod_2_dropout' #: 512 | |
| layer linear 'conformer_8_ffmod_2_dropout_linear' #: 512 | |
| layer eval 'conformer_8_ffmod_2_half_res_add' #: 512 | |
| layer linear 'conformer_8_ffmod_2_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_8_ffmod_2_ln' #: 512 | |
| layer linear 'conformer_8_mhsa_mod_att_linear' #: 512 | |
| layer copy 'conformer_8_mhsa_mod_dropout' #: 512 | |
| layer layer_norm 'conformer_8_mhsa_mod_ln' #: 512 | |
| layer relative_positional_encoding 'conformer_8_mhsa_mod_relpos_encoding' #: 64 | |
| layer combine 'conformer_8_mhsa_mod_res_add' #: 512 | |
| layer self_attention 'conformer_8_mhsa_mod_self_attention' #: 512 | |
| layer layer_norm 'conformer_8_output' #: 512 | |
| layer batch_norm 'conformer_9_conv_mod_bn' #: 512 | |
| layer conv 'conformer_9_conv_mod_depthwise_conv' #: 512 | |
| layer copy 'conformer_9_conv_mod_dropout' #: 512 | |
| layer gating 'conformer_9_conv_mod_glu' #: 512 | |
| layer layer_norm 'conformer_9_conv_mod_ln' #: 512 | |
| layer linear 'conformer_9_conv_mod_pointwise_conv_1' #: 1024 | |
| layer linear 'conformer_9_conv_mod_pointwise_conv_2' #: 512 | |
| layer combine 'conformer_9_conv_mod_res_add' #: 512 | |
| layer activation 'conformer_9_conv_mod_swish' #: 512 | |
| layer copy 'conformer_9_ffmod_1_dropout' #: 512 | |
| layer linear 'conformer_9_ffmod_1_dropout_linear' #: 512 | |
| layer eval 'conformer_9_ffmod_1_half_res_add' #: 512 | |
| layer linear 'conformer_9_ffmod_1_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_9_ffmod_1_ln' #: 512 | |
| layer copy 'conformer_9_ffmod_2_dropout' #: 512 | |
| layer linear 'conformer_9_ffmod_2_dropout_linear' #: 512 | |
| layer eval 'conformer_9_ffmod_2_half_res_add' #: 512 | |
| layer linear 'conformer_9_ffmod_2_linear_swish' #: 2048 | |
| layer layer_norm 'conformer_9_ffmod_2_ln' #: 512 | |
| layer linear 'conformer_9_mhsa_mod_att_linear' #: 512 | |
| layer copy 'conformer_9_mhsa_mod_dropout' #: 512 | |
| layer layer_norm 'conformer_9_mhsa_mod_ln' #: 512 | |
| layer relative_positional_encoding 'conformer_9_mhsa_mod_relpos_encoding' #: 64 | |
| layer combine 'conformer_9_mhsa_mod_res_add' #: 512 | |
| layer self_attention 'conformer_9_mhsa_mod_self_attention' #: 512 | |
| layer layer_norm 'conformer_9_output' #: 512 | |
| layer conv 'conv_1' #: 32 | |
| layer pool 'conv_1_pool' #: 32 | |
| layer conv 'conv_2' #: 64 | |
| layer conv 'conv_3' #: 64 | |
| layer merge_dims 'conv_merged' #: 24000 | |
| layer split_dims 'conv_source' #: 1 | |
| layer source 'data' #: 1 | |
| layer copy 'encoder' #: 512 | |
| layer subnetwork 'features' #: 750 | |
| layer conv 'features/conv_h' #: 150 | |
| layer eval 'features/conv_h_act' #: 150 | |
| layer variable 'features/conv_h_filter' #: 150 | |
| layer split_dims 'features/conv_h_split' #: 1 | |
| layer conv 'features/conv_l' #: 5 | |
| layer layer_norm 'features/conv_l_act' #: 750 | |
| layer eval 'features/conv_l_act_no_norm' #: 750 | |
| layer merge_dims 'features/conv_l_merge' #: 750 | |
| layer copy 'features/output' #: 750 | |
| layer copy 'input_dropout' #: 512 | |
| layer linear 'input_linear' #: 512 | |
| layer softmax 'output' #: 88 | |
| layer eval 'specaug' #: 750 | |
| net params #: 85180092 | |
| net trainable params: [<tf.Variable 'conformer_10_conv_mod_bn/batch_norm/conformer_10_conv_mod_bn_conformer_10_conv_mod_bn_output_beta:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_10_conv_mod_bn/batch_norm/conformer_10_conv_mod_bn_conformer_10_conv_mod_bn_output_gamma:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_10_conv_mod_depthwise_conv/W:0' shape=(32, 1, 512) dtype=float32>, <tf.Variable 'conformer_10_conv_mod_depthwise_conv/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_10_conv_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_10_conv_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_10_conv_mod_pointwise_conv_1/W:0' shape=(512, 1024) dtype=float32>, <tf.Variable 'conformer_10_conv_mod_pointwise_conv_1/b:0' shape=(1024,) dtype=float32>, <tf.Variable 'conformer_10_conv_mod_pointwise_conv_2/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_10_conv_mod_pointwise_conv_2/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_10_ffmod_1_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_10_ffmod_1_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_10_ffmod_1_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_10_ffmod_1_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_10_ffmod_1_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_10_ffmod_1_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_10_ffmod_2_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_10_ffmod_2_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_10_ffmod_2_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_10_ffmod_2_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_10_ffmod_2_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_10_ffmod_2_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_10_mhsa_mod_att_linear/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_10_mhsa_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_10_mhsa_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_10_mhsa_mod_relpos_encoding/encoding_matrix:0' shape=(65, 64) dtype=float32>, <tf.Variable 'conformer_10_mhsa_mod_self_attention/QKV:0' shape=(512, 1536) dtype=float32>, <tf.Variable 'conformer_10_output/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_10_output/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_11_conv_mod_bn/batch_norm/conformer_11_conv_mod_bn_conformer_11_conv_mod_bn_output_beta:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_11_conv_mod_bn/batch_norm/conformer_11_conv_mod_bn_conformer_11_conv_mod_bn_output_gamma:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_11_conv_mod_depthwise_conv/W:0' shape=(32, 1, 512) dtype=float32>, <tf.Variable 'conformer_11_conv_mod_depthwise_conv/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_11_conv_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_11_conv_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_11_conv_mod_pointwise_conv_1/W:0' shape=(512, 1024) dtype=float32>, <tf.Variable 'conformer_11_conv_mod_pointwise_conv_1/b:0' shape=(1024,) dtype=float32>, <tf.Variable 'conformer_11_conv_mod_pointwise_conv_2/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_11_conv_mod_pointwise_conv_2/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_11_ffmod_1_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_11_ffmod_1_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_11_ffmod_1_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_11_ffmod_1_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_11_ffmod_1_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_11_ffmod_1_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_11_ffmod_2_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_11_ffmod_2_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_11_ffmod_2_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_11_ffmod_2_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_11_ffmod_2_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_11_ffmod_2_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_11_mhsa_mod_att_linear/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_11_mhsa_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_11_mhsa_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_11_mhsa_mod_relpos_encoding/encoding_matrix:0' shape=(65, 64) dtype=float32>, <tf.Variable 'conformer_11_mhsa_mod_self_attention/QKV:0' shape=(512, 1536) dtype=float32>, <tf.Variable 'conformer_11_output/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_11_output/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_12_conv_mod_bn/batch_norm/conformer_12_conv_mod_bn_conformer_12_conv_mod_bn_output_beta:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_12_conv_mod_bn/batch_norm/conformer_12_conv_mod_bn_conformer_12_conv_mod_bn_output_gamma:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_12_conv_mod_depthwise_conv/W:0' shape=(32, 1, 512) dtype=float32>, <tf.Variable 'conformer_12_conv_mod_depthwise_conv/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_12_conv_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_12_conv_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_12_conv_mod_pointwise_conv_1/W:0' shape=(512, 1024) dtype=float32>, <tf.Variable 'conformer_12_conv_mod_pointwise_conv_1/b:0' shape=(1024,) dtype=float32>, <tf.Variable 'conformer_12_conv_mod_pointwise_conv_2/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_12_conv_mod_pointwise_conv_2/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_12_ffmod_1_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_12_ffmod_1_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_12_ffmod_1_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_12_ffmod_1_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_12_ffmod_1_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_12_ffmod_1_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_12_ffmod_2_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_12_ffmod_2_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_12_ffmod_2_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_12_ffmod_2_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_12_ffmod_2_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_12_ffmod_2_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_12_mhsa_mod_att_linear/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_12_mhsa_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_12_mhsa_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_12_mhsa_mod_relpos_encoding/encoding_matrix:0' shape=(65, 64) dtype=float32>, <tf.Variable 'conformer_12_mhsa_mod_self_attention/QKV:0' shape=(512, 1536) dtype=float32>, <tf.Variable 'conformer_12_output/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_12_output/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_1_conv_mod_bn/batch_norm/conformer_1_conv_mod_bn_conformer_1_conv_mod_bn_output_beta:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_1_conv_mod_bn/batch_norm/conformer_1_conv_mod_bn_conformer_1_conv_mod_bn_output_gamma:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_1_conv_mod_depthwise_conv/W:0' shape=(32, 1, 512) dtype=float32>, <tf.Variable 'conformer_1_conv_mod_depthwise_conv/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_1_conv_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_1_conv_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_1_conv_mod_pointwise_conv_1/W:0' shape=(512, 1024) dtype=float32>, <tf.Variable 'conformer_1_conv_mod_pointwise_conv_1/b:0' shape=(1024,) dtype=float32>, <tf.Variable 'conformer_1_conv_mod_pointwise_conv_2/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_1_conv_mod_pointwise_conv_2/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_1_ffmod_1_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_1_ffmod_1_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_1_ffmod_1_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_1_ffmod_1_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_1_ffmod_1_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_1_ffmod_1_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_1_ffmod_2_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_1_ffmod_2_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_1_ffmod_2_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_1_ffmod_2_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_1_ffmod_2_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_1_ffmod_2_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_1_mhsa_mod_att_linear/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_1_mhsa_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_1_mhsa_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_1_mhsa_mod_relpos_encoding/encoding_matrix:0' shape=(65, 64) dtype=float32>, <tf.Variable 'conformer_1_mhsa_mod_self_attention/QKV:0' shape=(512, 1536) dtype=float32>, <tf.Variable 'conformer_1_output/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_1_output/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_2_conv_mod_bn/batch_norm/conformer_2_conv_mod_bn_conformer_2_conv_mod_bn_output_beta:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_2_conv_mod_bn/batch_norm/conformer_2_conv_mod_bn_conformer_2_conv_mod_bn_output_gamma:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_2_conv_mod_depthwise_conv/W:0' shape=(32, 1, 512) dtype=float32>, <tf.Variable 'conformer_2_conv_mod_depthwise_conv/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_2_conv_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_2_conv_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_2_conv_mod_pointwise_conv_1/W:0' shape=(512, 1024) dtype=float32>, <tf.Variable 'conformer_2_conv_mod_pointwise_conv_1/b:0' shape=(1024,) dtype=float32>, <tf.Variable 'conformer_2_conv_mod_pointwise_conv_2/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_2_conv_mod_pointwise_conv_2/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_2_ffmod_1_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_2_ffmod_1_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_2_ffmod_1_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_2_ffmod_1_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_2_ffmod_1_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_2_ffmod_1_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_2_ffmod_2_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_2_ffmod_2_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_2_ffmod_2_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_2_ffmod_2_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_2_ffmod_2_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_2_ffmod_2_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_2_mhsa_mod_att_linear/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_2_mhsa_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_2_mhsa_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_2_mhsa_mod_relpos_encoding/encoding_matrix:0' shape=(65, 64) dtype=float32>, <tf.Variable 'conformer_2_mhsa_mod_self_attention/QKV:0' shape=(512, 1536) dtype=float32>, <tf.Variable 'conformer_2_output/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_2_output/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_3_conv_mod_bn/batch_norm/conformer_3_conv_mod_bn_conformer_3_conv_mod_bn_output_beta:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_3_conv_mod_bn/batch_norm/conformer_3_conv_mod_bn_conformer_3_conv_mod_bn_output_gamma:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_3_conv_mod_depthwise_conv/W:0' shape=(32, 1, 512) dtype=float32>, <tf.Variable 'conformer_3_conv_mod_depthwise_conv/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_3_conv_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_3_conv_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_3_conv_mod_pointwise_conv_1/W:0' shape=(512, 1024) dtype=float32>, <tf.Variable 'conformer_3_conv_mod_pointwise_conv_1/b:0' shape=(1024,) dtype=float32>, <tf.Variable 'conformer_3_conv_mod_pointwise_conv_2/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_3_conv_mod_pointwise_conv_2/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_3_ffmod_1_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_3_ffmod_1_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_3_ffmod_1_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_3_ffmod_1_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_3_ffmod_1_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_3_ffmod_1_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_3_ffmod_2_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_3_ffmod_2_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_3_ffmod_2_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_3_ffmod_2_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_3_ffmod_2_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_3_ffmod_2_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_3_mhsa_mod_att_linear/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_3_mhsa_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_3_mhsa_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_3_mhsa_mod_relpos_encoding/encoding_matrix:0' shape=(65, 64) dtype=float32>, <tf.Variable 'conformer_3_mhsa_mod_self_attention/QKV:0' shape=(512, 1536) dtype=float32>, <tf.Variable 'conformer_3_output/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_3_output/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_4_conv_mod_bn/batch_norm/conformer_4_conv_mod_bn_conformer_4_conv_mod_bn_output_beta:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_4_conv_mod_bn/batch_norm/conformer_4_conv_mod_bn_conformer_4_conv_mod_bn_output_gamma:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_4_conv_mod_depthwise_conv/W:0' shape=(32, 1, 512) dtype=float32>, <tf.Variable 'conformer_4_conv_mod_depthwise_conv/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_4_conv_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_4_conv_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_4_conv_mod_pointwise_conv_1/W:0' shape=(512, 1024) dtype=float32>, <tf.Variable 'conformer_4_conv_mod_pointwise_conv_1/b:0' shape=(1024,) dtype=float32>, <tf.Variable 'conformer_4_conv_mod_pointwise_conv_2/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_4_conv_mod_pointwise_conv_2/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_4_ffmod_1_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_4_ffmod_1_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_4_ffmod_1_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_4_ffmod_1_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_4_ffmod_1_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_4_ffmod_1_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_4_ffmod_2_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_4_ffmod_2_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_4_ffmod_2_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_4_ffmod_2_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_4_ffmod_2_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_4_ffmod_2_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_4_mhsa_mod_att_linear/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_4_mhsa_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_4_mhsa_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_4_mhsa_mod_relpos_encoding/encoding_matrix:0' shape=(65, 64) dtype=float32>, <tf.Variable 'conformer_4_mhsa_mod_self_attention/QKV:0' shape=(512, 1536) dtype=float32>, <tf.Variable 'conformer_4_output/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_4_output/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_5_conv_mod_bn/batch_norm/conformer_5_conv_mod_bn_conformer_5_conv_mod_bn_output_beta:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_5_conv_mod_bn/batch_norm/conformer_5_conv_mod_bn_conformer_5_conv_mod_bn_output_gamma:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_5_conv_mod_depthwise_conv/W:0' shape=(32, 1, 512) dtype=float32>, <tf.Variable 'conformer_5_conv_mod_depthwise_conv/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_5_conv_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_5_conv_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_5_conv_mod_pointwise_conv_1/W:0' shape=(512, 1024) dtype=float32>, <tf.Variable 'conformer_5_conv_mod_pointwise_conv_1/b:0' shape=(1024,) dtype=float32>, <tf.Variable 'conformer_5_conv_mod_pointwise_conv_2/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_5_conv_mod_pointwise_conv_2/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_5_ffmod_1_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_5_ffmod_1_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_5_ffmod_1_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_5_ffmod_1_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_5_ffmod_1_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_5_ffmod_1_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_5_ffmod_2_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_5_ffmod_2_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_5_ffmod_2_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_5_ffmod_2_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_5_ffmod_2_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_5_ffmod_2_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_5_mhsa_mod_att_linear/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_5_mhsa_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_5_mhsa_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_5_mhsa_mod_relpos_encoding/encoding_matrix:0' shape=(65, 64) dtype=float32>, <tf.Variable 'conformer_5_mhsa_mod_self_attention/QKV:0' shape=(512, 1536) dtype=float32>, <tf.Variable 'conformer_5_output/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_5_output/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_6_conv_mod_bn/batch_norm/conformer_6_conv_mod_bn_conformer_6_conv_mod_bn_output_beta:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_6_conv_mod_bn/batch_norm/conformer_6_conv_mod_bn_conformer_6_conv_mod_bn_output_gamma:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_6_conv_mod_depthwise_conv/W:0' shape=(32, 1, 512) dtype=float32>, <tf.Variable 'conformer_6_conv_mod_depthwise_conv/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_6_conv_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_6_conv_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_6_conv_mod_pointwise_conv_1/W:0' shape=(512, 1024) dtype=float32>, <tf.Variable 'conformer_6_conv_mod_pointwise_conv_1/b:0' shape=(1024,) dtype=float32>, <tf.Variable 'conformer_6_conv_mod_pointwise_conv_2/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_6_conv_mod_pointwise_conv_2/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_6_ffmod_1_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_6_ffmod_1_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_6_ffmod_1_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_6_ffmod_1_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_6_ffmod_1_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_6_ffmod_1_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_6_ffmod_2_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_6_ffmod_2_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_6_ffmod_2_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_6_ffmod_2_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_6_ffmod_2_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_6_ffmod_2_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_6_mhsa_mod_att_linear/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_6_mhsa_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_6_mhsa_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_6_mhsa_mod_relpos_encoding/encoding_matrix:0' shape=(65, 64) dtype=float32>, <tf.Variable 'conformer_6_mhsa_mod_self_attention/QKV:0' shape=(512, 1536) dtype=float32>, <tf.Variable 'conformer_6_output/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_6_output/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_7_conv_mod_bn/batch_norm/conformer_7_conv_mod_bn_conformer_7_conv_mod_bn_output_beta:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_7_conv_mod_bn/batch_norm/conformer_7_conv_mod_bn_conformer_7_conv_mod_bn_output_gamma:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_7_conv_mod_depthwise_conv/W:0' shape=(32, 1, 512) dtype=float32>, <tf.Variable 'conformer_7_conv_mod_depthwise_conv/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_7_conv_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_7_conv_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_7_conv_mod_pointwise_conv_1/W:0' shape=(512, 1024) dtype=float32>, <tf.Variable 'conformer_7_conv_mod_pointwise_conv_1/b:0' shape=(1024,) dtype=float32>, <tf.Variable 'conformer_7_conv_mod_pointwise_conv_2/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_7_conv_mod_pointwise_conv_2/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_7_ffmod_1_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_7_ffmod_1_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_7_ffmod_1_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_7_ffmod_1_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_7_ffmod_1_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_7_ffmod_1_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_7_ffmod_2_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_7_ffmod_2_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_7_ffmod_2_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_7_ffmod_2_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_7_ffmod_2_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_7_ffmod_2_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_7_mhsa_mod_att_linear/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_7_mhsa_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_7_mhsa_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_7_mhsa_mod_relpos_encoding/encoding_matrix:0' shape=(65, 64) dtype=float32>, <tf.Variable 'conformer_7_mhsa_mod_self_attention/QKV:0' shape=(512, 1536) dtype=float32>, <tf.Variable 'conformer_7_output/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_7_output/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_8_conv_mod_bn/batch_norm/conformer_8_conv_mod_bn_conformer_8_conv_mod_bn_output_beta:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_8_conv_mod_bn/batch_norm/conformer_8_conv_mod_bn_conformer_8_conv_mod_bn_output_gamma:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_8_conv_mod_depthwise_conv/W:0' shape=(32, 1, 512) dtype=float32>, <tf.Variable 'conformer_8_conv_mod_depthwise_conv/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_8_conv_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_8_conv_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_8_conv_mod_pointwise_conv_1/W:0' shape=(512, 1024) dtype=float32>, <tf.Variable 'conformer_8_conv_mod_pointwise_conv_1/b:0' shape=(1024,) dtype=float32>, <tf.Variable 'conformer_8_conv_mod_pointwise_conv_2/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_8_conv_mod_pointwise_conv_2/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_8_ffmod_1_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_8_ffmod_1_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_8_ffmod_1_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_8_ffmod_1_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_8_ffmod_1_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_8_ffmod_1_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_8_ffmod_2_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_8_ffmod_2_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_8_ffmod_2_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_8_ffmod_2_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_8_ffmod_2_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_8_ffmod_2_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_8_mhsa_mod_att_linear/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_8_mhsa_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_8_mhsa_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_8_mhsa_mod_relpos_encoding/encoding_matrix:0' shape=(65, 64) dtype=float32>, <tf.Variable 'conformer_8_mhsa_mod_self_attention/QKV:0' shape=(512, 1536) dtype=float32>, <tf.Variable 'conformer_8_output/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_8_output/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_9_conv_mod_bn/batch_norm/conformer_9_conv_mod_bn_conformer_9_conv_mod_bn_output_beta:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_9_conv_mod_bn/batch_norm/conformer_9_conv_mod_bn_conformer_9_conv_mod_bn_output_gamma:0' shape=(1, 1, 512) dtype=float32>, <tf.Variable 'conformer_9_conv_mod_depthwise_conv/W:0' shape=(32, 1, 512) dtype=float32>, <tf.Variable 'conformer_9_conv_mod_depthwise_conv/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_9_conv_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_9_conv_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_9_conv_mod_pointwise_conv_1/W:0' shape=(512, 1024) dtype=float32>, <tf.Variable 'conformer_9_conv_mod_pointwise_conv_1/b:0' shape=(1024,) dtype=float32>, <tf.Variable 'conformer_9_conv_mod_pointwise_conv_2/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_9_conv_mod_pointwise_conv_2/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_9_ffmod_1_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_9_ffmod_1_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_9_ffmod_1_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_9_ffmod_1_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_9_ffmod_1_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_9_ffmod_1_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_9_ffmod_2_dropout_linear/W:0' shape=(2048, 512) dtype=float32>, <tf.Variable 'conformer_9_ffmod_2_dropout_linear/b:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_9_ffmod_2_linear_swish/W:0' shape=(512, 2048) dtype=float32>, <tf.Variable 'conformer_9_ffmod_2_linear_swish/b:0' shape=(2048,) dtype=float32>, <tf.Variable 'conformer_9_ffmod_2_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_9_ffmod_2_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_9_mhsa_mod_att_linear/W:0' shape=(512, 512) dtype=float32>, <tf.Variable 'conformer_9_mhsa_mod_ln/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_9_mhsa_mod_ln/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_9_mhsa_mod_relpos_encoding/encoding_matrix:0' shape=(65, 64) dtype=float32>, <tf.Variable 'conformer_9_mhsa_mod_self_attention/QKV:0' shape=(512, 1536) dtype=float32>, <tf.Variable 'conformer_9_output/bias:0' shape=(512,) dtype=float32>, <tf.Variable 'conformer_9_output/scale:0' shape=(512,) dtype=float32>, <tf.Variable 'conv_1/W:0' shape=(3, 3, 1, 32) dtype=float32>, <tf.Variable 'conv_1/bias:0' shape=(32,) dtype=float32>, <tf.Variable 'conv_2/W:0' shape=(3, 3, 32, 64) dtype=float32>, <tf.Variable 'conv_2/bias:0' shape=(64,) dtype=float32>, <tf.Variable 'conv_3/W:0' shape=(3, 3, 64, 64) dtype=float32>, <tf.Variable 'conv_3/bias:0' shape=(64,) dtype=float32>, <tf.Variable 'features/conv_h_filter/conv_h_filter:0' shape=(128, 1, 150) dtype=float32>, <tf.Variable 'features/conv_l/W:0' shape=(40, 1, 1, 5) dtype=float32>, <tf.Variable 'features/conv_l_act/bias:0' shape=(750,) dtype=float32>, <tf.Variable 'features/conv_l_act/scale:0' shape=(750,) dtype=float32>, <tf.Variable 'input_linear/W:0' shape=(24000, 512) dtype=float32>, <tf.Variable 'output/W:0' shape=(512, 88) dtype=float32>, <tf.Variable 'output/b:0' shape=(88,) dtype=float32>] | |
| MEMORY: main proc python3(1781720) increased RSS: rss=1.5GB pss=1.5GB uss=1.5GB shared=29.0MB | |
| MEMORY: total (5 procs): pss=2.1GB uss=2.0GB | |
| loading weights from /u/maximilian.kannen/setups/20230406_feat/work/i6_core/returnn/training/ReturnnTrainingJob.wQIA3Rc1nLak/output/models/epoch.062 | |
| [2023-10-31 00:05:19,989] INFO: [32mRun time: 0:00:40 CPU: 0.20% RSS: 2.88GB VMS: 31.76GB[0m | |
| start training at epoch 63 | |
| using batch size: {'classes': 5000, 'data': 400000}, max seqs: 128 | |
| learning rate control: NewbobMultiEpoch(num_epochs=6, update_interval=1, relative_error_threshold=-0.01, relative_error_grow_threshold=-0.01), epoch data: 1: EpochData(learningRate=1.325e-05, error={ | |
| 'dev_score': 1.496220382764843, | |
| 'devtrain_score': 1.448848050936363, | |
| 'train_score': 1.3928855032347311, | |
| }), 2: EpochData(learningRate=1.539861111111111e-05, error={ | |
| 'dev_score': 1.3514346086708577, | |
| 'devtrain_score': 1.3205680706585257, | |
| 'train_score': 1.3352038787295408, | |
| }), 3: EpochData(learningRate=1.754722222222222e-05, error={ | |
| 'dev_score': 1.236011079403838, | |
| 'devtrain_score': 1.204534264175696, | |
| 'train_score': 1.2634383703273775, | |
| }), ..., 360: EpochData(learningRate=1.4333333333333375e-05, error={}), 361: EpochData(learningRate=1.2166666666666727e-05, error={}), 362: EpochData(learningRate=1e-05, error={}), error key: dev_score | |
| pretrain: None | |
| [2023-10-31 00:05:25,026] INFO: [32mRun time: 0:00:45 CPU: 0.20% RSS: 4.63GB VMS: 33.52GB[0m | |
| MEMORY: main proc python3(1781720) increased RSS: rss=2.8GB pss=2.7GB uss=2.7GB shared=31.7MB | |
| MEMORY: sub proc MPD seq order(1781781) increased RSS: rss=573.1MB pss=556.1MB uss=551.7MB shared=21.4MB | |
| MEMORY: sub proc MPD worker(1781782) increased RSS: rss=274.5MB pss=258.2MB uss=254.2MB shared=20.3MB | |
| MEMORY: sub proc MPD worker(1781783) increased RSS: rss=440.3MB pss=423.3MB uss=419.0MB shared=21.2MB | |
| MEMORY: total (5 procs): pss=3.9GB uss=3.9GB | |
| start epoch 63 with learning rate 0.00014646388888888887 ... | |
| Create optimizer <class 'returnn.tf.updater.NadamOptimizer'> with options {'epsilon': 1e-08, 'learning_rate': <tf.Variable 'learning_rate:0' shape=() dtype=float32>}. | |
| MEMORY: main proc python3(1781720) increased RSS: rss=2.9GB pss=2.9GB uss=2.9GB shared=31.8MB | |
| MEMORY: sub proc MPD worker(1781782) increased RSS: rss=573.2MB pss=555.5MB uss=550.9MB shared=22.3MB | |
| MEMORY: sub proc MPD worker(1781783) increased RSS: rss=573.3MB pss=555.5MB uss=550.9MB shared=22.4MB | |
| MEMORY: total (5 procs): pss=4.5GB uss=4.5GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=3.0GB pss=2.9GB uss=2.9GB shared=31.8MB | |
| MEMORY: total (5 procs): pss=4.6GB uss=4.5GB | |
| Initialize optimizer (default) with slots ['m', 'v']. | |
| These additional variable were created by the optimizer: [<tf.Variable 'optimize/beta1_power:0' shape=() dtype=float32>, <tf.Variable 'optimize/beta2_power:0' shape=() dtype=float32>]. | |
| /work/asr4/vieting/programs/rasr/20230707/rasr/arch/linux-x86_64-standard/nn-trainer.linux-x86_64-standard: Relink `/usr/local/lib/tensorflow/libtensorflow_framework.so.2' with `/lib/x86_64-linux-gnu/libz.so.1' for IFUNC symbol `crc32_z' | |
| [2023-10-31 00:06:00,126] INFO: [32mRun time: 0:01:20 CPU: 0.40% RSS: 5.32GB VMS: 35.32GB[0m | |
| configuration error: failed to open file "neural-network-trainer.config" for reading. (No such file or directory) | |
| /work/asr4/vieting/programs/rasr/20230707/rasr/arch/linux-x86_64-standard/nn-trainer.linux-x86_64-standard: Relink `/usr/local/lib/tensorflow/libtensorflow_framework.so.2' with `/lib/x86_64-linux-gnu/libz.so.1' for IFUNC symbol `crc32_z' | |
| configuration error: failed to open file "neural-network-trainer.config" for reading. (No such file or directory) | |
| [2023-10-31 00:06:15,155] INFO: [32mRun time: 0:01:35 CPU: 0.00% RSS: 6.62GB VMS: 38.06GB[0m | |
| 2023-10-31 00:06:16.013852: I tensorflow/stream_executor/cuda/cuda_dnn.cc:379] Loaded cuDNN version 8400 | |
| [2023-10-31 00:06:20,171] INFO: [32mRun time: 0:01:40 CPU: 0.20% RSS: 8.32GB VMS: 41.12GB[0m | |
| 2023-10-31 00:06:20.481813: I tensorflow/core/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory | |
| 2023-10-31 00:06:20.728126: I tensorflow/stream_executor/cuda/cuda_blas.cc:1804] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once. | |
| [2023-10-31 00:08:15,483] INFO: [32mRun time: 0:03:35 CPU: 0.20% RSS: 9.15GB VMS: 42.37GB[0m | |
| [2023-10-31 00:13:01,230] INFO: [32mRun time: 0:08:21 CPU: 0.40% RSS: 10.06GB VMS: 43.75GB[0m | |
| [2023-10-31 00:18:42,099] INFO: [32mRun time: 0:14:02 CPU: 0.20% RSS: 11.08GB VMS: 45.25GB[0m | |
| [2023-10-31 00:25:28,147] INFO: [32mRun time: 0:20:48 CPU: 0.40% RSS: 12.19GB VMS: 46.82GB[0m | |
| [2023-10-31 00:33:39,403] INFO: [32mRun time: 0:28:59 CPU: 0.20% RSS: 13.42GB VMS: 48.52GB[0m | |
| MEMORY: main proc python3(1781720) increased RSS: rss=3.2GB pss=3.2GB uss=3.2GB shared=31.8MB | |
| MEMORY: total (5 procs): pss=4.8GB uss=4.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=3.3GB pss=3.3GB uss=3.3GB shared=31.8MB | |
| MEMORY: total (5 procs): pss=4.9GB uss=4.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=3.5GB pss=3.5GB uss=3.5GB shared=38.7MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) initial: rss=47.1MB pss=41.4MB uss=37.0MB shared=10.1MB | |
| MEMORY: total (6 procs): pss=5.2GB uss=5.1GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=3.7GB pss=3.7GB uss=3.7GB shared=46.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=94.5MB pss=82.5MB uss=72.8MB shared=21.7MB | |
| MEMORY: total (6 procs): pss=5.4GB uss=5.3GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=3.8GB pss=3.8GB uss=3.8GB shared=46.5MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=126.5MB pss=114.3MB uss=104.6MB shared=21.9MB | |
| MEMORY: total (6 procs): pss=5.5GB uss=5.5GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=4.6GB pss=4.6GB uss=4.6GB shared=48.0MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=222.7MB pss=150.9MB uss=98.9MB shared=123.8MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) initial: rss=226.4MB pss=154.3MB uss=102.0MB shared=124.5MB | |
| MEMORY: total (7 procs): pss=6.5GB uss=6.4GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=6.2GB pss=6.2GB uss=6.2GB shared=48.1MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=271.5MB pss=198.4MB uss=145.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=273.2MB pss=199.8MB uss=147.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=8.2GB uss=8.1GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=6.5GB pss=6.4GB uss=6.4GB shared=48.1MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=271.6MB pss=198.6MB uss=146.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=273.4MB pss=200.0MB uss=147.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=8.5GB uss=8.3GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=6.6GB pss=6.5GB uss=6.5GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=8.6GB uss=8.4GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=6.7GB pss=6.6GB uss=6.6GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=8.6GB uss=8.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=271.7MB pss=198.6MB uss=146.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=8.6GB uss=8.5GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=6.8GB pss=6.7GB uss=6.7GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=8.8GB uss=8.6GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=6.9GB pss=6.8GB uss=6.8GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=8.8GB uss=8.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=271.8MB pss=198.8MB uss=146.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=8.8GB uss=8.7GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=7.0GB pss=6.9GB uss=6.9GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=8.9GB uss=8.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=271.9MB pss=198.8MB uss=146.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=8.9GB uss=8.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=273.6MB pss=200.2MB uss=147.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=8.9GB uss=8.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=272.0MB pss=198.9MB uss=146.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=273.7MB pss=200.3MB uss=147.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=8.9GB uss=8.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=7.1GB pss=7.0GB uss=7.0GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=9.0GB uss=8.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=7.2GB pss=7.1GB uss=7.1GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=9.1GB uss=9.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=273.8MB pss=200.4MB uss=147.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=9.1GB uss=9.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=272.1MB pss=199.0MB uss=146.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=9.1GB uss=9.0GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=7.3GB pss=7.2GB uss=7.2GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=9.3GB uss=9.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=272.4MB pss=199.3MB uss=146.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=9.3GB uss=9.1GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=7.4GB pss=7.3GB uss=7.3GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=9.3GB uss=9.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=273.9MB pss=200.5MB uss=147.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=9.3GB uss=9.2GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=7.5GB pss=7.4GB uss=7.4GB shared=48.1MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=274.1MB pss=200.8MB uss=148.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=9.5GB uss=9.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=274.2MB pss=200.8MB uss=148.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=9.5GB uss=9.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=272.5MB pss=199.4MB uss=146.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=9.5GB uss=9.3GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=7.6GB pss=7.5GB uss=7.5GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=9.5GB uss=9.4GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=7.7GB pss=7.6GB uss=7.6GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=9.6GB uss=9.5GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=7.8GB pss=7.7GB uss=7.7GB shared=48.1MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=272.6MB pss=199.5MB uss=147.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=9.7GB uss=9.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=274.3MB pss=200.9MB uss=148.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=9.7GB uss=9.6GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=7.9GB pss=7.8GB uss=7.8GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=9.8GB uss=9.7GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=8.0GB pss=7.9GB uss=7.9GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=10.0GB uss=9.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=272.7MB pss=199.6MB uss=147.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=10.0GB uss=9.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=8.1GB pss=8.0GB uss=8.0GB shared=48.1MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=274.4MB pss=201.0MB uss=148.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=10.0GB uss=9.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=273.3MB pss=200.2MB uss=147.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=10.0GB uss=9.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=273.6MB pss=200.5MB uss=148.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=10.0GB uss=9.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=8.2GB pss=8.1GB uss=8.1GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=10.1GB uss=10.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=275.0MB pss=201.6MB uss=148.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=10.1GB uss=10.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=275.3MB pss=201.9MB uss=149.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=10.1GB uss=10.0GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=8.3GB pss=8.2GB uss=8.2GB shared=48.1MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=273.7MB pss=200.6MB uss=148.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=10.3GB uss=10.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=275.4MB pss=202.0MB uss=149.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=10.3GB uss=10.1GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=8.4GB pss=8.3GB uss=8.3GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=10.4GB uss=10.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=273.8MB pss=200.7MB uss=148.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=10.4GB uss=10.2GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=8.5GB pss=8.4GB uss=8.4GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=10.4GB uss=10.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=273.9MB pss=200.8MB uss=148.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=275.5MB pss=202.1MB uss=149.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=10.4GB uss=10.3GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=8.6GB pss=8.5GB uss=8.5GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=10.6GB uss=10.4GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=8.7GB pss=8.6GB uss=8.6GB shared=48.1MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=274.0MB pss=200.9MB uss=148.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=275.6MB pss=202.2MB uss=149.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=10.7GB uss=10.5GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=8.8GB pss=8.7GB uss=8.7GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=10.7GB uss=10.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=274.1MB pss=201.0MB uss=148.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=275.7MB pss=202.3MB uss=149.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=10.8GB uss=10.6GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=8.9GB pss=8.8GB uss=8.8GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=10.9GB uss=10.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=274.2MB pss=201.1MB uss=148.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=10.9GB uss=10.7GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=9.0GB pss=8.9GB uss=8.9GB shared=48.1MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=275.8MB pss=202.4MB uss=149.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=11.0GB uss=10.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=9.1GB pss=9.0GB uss=9.0GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=11.0GB uss=10.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=274.3MB pss=201.2MB uss=148.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=11.0GB uss=10.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=275.9MB pss=202.5MB uss=149.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=11.0GB uss=10.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=9.2GB pss=9.1GB uss=9.1GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=11.2GB uss=11.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=274.4MB pss=201.3MB uss=148.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=11.2GB uss=11.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=276.0MB pss=202.6MB uss=149.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=11.2GB uss=11.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=275.0MB pss=201.9MB uss=149.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=11.2GB uss=11.0GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=9.3GB pss=9.2GB uss=9.2GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=11.3GB uss=11.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=275.1MB pss=202.0MB uss=149.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=11.3GB uss=11.1GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=9.4GB pss=9.3GB uss=9.3GB shared=48.1MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=276.1MB pss=202.7MB uss=149.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=11.4GB uss=11.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=275.2MB pss=202.1MB uss=149.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=11.4GB uss=11.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=276.2MB pss=202.8MB uss=150.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=11.4GB uss=11.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=276.8MB pss=203.4MB uss=150.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=11.4GB uss=11.2GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=9.5GB pss=9.4GB uss=9.4GB shared=48.1MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=275.3MB pss=202.2MB uss=149.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=11.5GB uss=11.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=276.9MB pss=203.5MB uss=150.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=11.5GB uss=11.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=275.4MB pss=202.3MB uss=149.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=11.5GB uss=11.3GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=9.6GB pss=9.5GB uss=9.5GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=11.5GB uss=11.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=277.0MB pss=203.6MB uss=150.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=11.5GB uss=11.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=275.5MB pss=202.4MB uss=149.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=11.5GB uss=11.4GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=9.7GB pss=9.6GB uss=9.6GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=11.6GB uss=11.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=277.1MB pss=203.7MB uss=150.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=11.6GB uss=11.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=275.6MB pss=202.5MB uss=150.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=11.6GB uss=11.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=276.1MB pss=203.1MB uss=150.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=11.7GB uss=11.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=277.2MB pss=203.8MB uss=151.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=11.7GB uss=11.5GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=9.8GB pss=9.7GB uss=9.7GB shared=48.1MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=276.2MB pss=203.1MB uss=150.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=11.7GB uss=11.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=276.3MB pss=203.2MB uss=150.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=277.7MB pss=204.4MB uss=151.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=11.7GB uss=11.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=277.8MB pss=204.4MB uss=151.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=11.7GB uss=11.6GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=9.9GB pss=9.8GB uss=9.8GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=11.9GB uss=11.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=276.4MB pss=203.3MB uss=150.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=11.9GB uss=11.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=277.9MB pss=204.5MB uss=151.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=11.9GB uss=11.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=276.5MB pss=203.4MB uss=150.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=11.9GB uss=11.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=278.0MB pss=204.6MB uss=151.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=11.9GB uss=11.7GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=10.0GB pss=9.9GB uss=9.9GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=12.0GB uss=11.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=276.6MB pss=203.5MB uss=151.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=12.0GB uss=11.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=276.7MB pss=203.7MB uss=151.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=278.1MB pss=204.7MB uss=151.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=12.0GB uss=11.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=276.8MB pss=203.7MB uss=151.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=12.0GB uss=11.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=278.2MB pss=204.8MB uss=152.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=12.0GB uss=11.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=10.1GB pss=10.0GB uss=10.0GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=12.1GB uss=11.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=276.9MB pss=203.8MB uss=151.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=12.1GB uss=11.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=277.0MB pss=203.9MB uss=151.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=278.3MB pss=205.0MB uss=152.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=12.1GB uss=11.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=277.1MB pss=204.0MB uss=151.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=12.1GB uss=11.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=278.4MB pss=205.0MB uss=152.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=12.1GB uss=11.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=10.2GB pss=10.1GB uss=10.1GB shared=48.1MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=277.2MB pss=204.1MB uss=151.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=12.2GB uss=12.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=277.3MB pss=204.2MB uss=151.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=12.2GB uss=12.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=277.8MB pss=204.7MB uss=152.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=12.2GB uss=12.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=277.9MB pss=204.8MB uss=152.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=278.5MB pss=205.1MB uss=152.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=12.2GB uss=12.0GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=10.3GB pss=10.2GB uss=10.2GB shared=48.1MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=278.0MB pss=204.9MB uss=152.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=12.3GB uss=12.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=278.6MB pss=205.2MB uss=152.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=12.3GB uss=12.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=278.1MB pss=205.0MB uss=152.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=278.7MB pss=205.3MB uss=152.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=12.3GB uss=12.1GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=10.4GB pss=10.3GB uss=10.3GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=12.4GB uss=12.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=278.8MB pss=205.4MB uss=152.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=12.4GB uss=12.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=278.2MB pss=205.1MB uss=152.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=278.9MB pss=205.5MB uss=152.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=12.4GB uss=12.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=279.0MB pss=205.6MB uss=152.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=12.4GB uss=12.2GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=10.5GB pss=10.4GB uss=10.4GB shared=48.1MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=278.3MB pss=205.3MB uss=152.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=280.0MB pss=206.6MB uss=153.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=12.5GB uss=12.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=278.4MB pss=205.3MB uss=152.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=12.5GB uss=12.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=278.5MB pss=205.4MB uss=152.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=12.5GB uss=12.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=280.1MB pss=206.7MB uss=153.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=12.5GB uss=12.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=278.7MB pss=205.6MB uss=153.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=12.5GB uss=12.3GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=10.6GB pss=10.5GB uss=10.5GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=12.6GB uss=12.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=278.8MB pss=205.7MB uss=153.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=12.6GB uss=12.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=278.9MB pss=205.9MB uss=153.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=280.2MB pss=206.8MB uss=154.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=12.6GB uss=12.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=279.4MB pss=206.4MB uss=153.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=12.6GB uss=12.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=279.5MB pss=206.4MB uss=153.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=12.6GB uss=12.4GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=10.7GB pss=10.6GB uss=10.6GB shared=48.1MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=279.9MB pss=206.9MB uss=154.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=12.7GB uss=12.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=280.0MB pss=206.9MB uss=154.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=280.3MB pss=207.0MB uss=154.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=12.7GB uss=12.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=280.4MB pss=207.0MB uss=154.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=12.7GB uss=12.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=280.1MB pss=207.0MB uss=154.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=280.5MB pss=207.1MB uss=154.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=12.7GB uss=12.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=280.6MB pss=207.2MB uss=154.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=12.7GB uss=12.5GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=10.8GB pss=10.7GB uss=10.7GB shared=48.1MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=281.1MB pss=207.7MB uss=154.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=12.8GB uss=12.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=280.2MB pss=207.1MB uss=154.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=12.8GB uss=12.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=281.2MB pss=207.8MB uss=155.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=12.8GB uss=12.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=280.3MB pss=207.2MB uss=154.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=12.8GB uss=12.6GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=10.9GB pss=10.8GB uss=10.8GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=12.9GB uss=12.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=280.4MB pss=207.3MB uss=154.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=12.9GB uss=12.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=281.3MB pss=207.9MB uss=155.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=12.9GB uss=12.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=280.5MB pss=207.4MB uss=154.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=12.9GB uss=12.7GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=11.0GB pss=10.9GB uss=10.9GB shared=48.1MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=280.6MB pss=207.5MB uss=155.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=281.4MB pss=208.0MB uss=155.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.0GB uss=12.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=281.5MB pss=208.1MB uss=155.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.0GB uss=12.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=280.7MB pss=207.6MB uss=155.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=281.6MB pss=208.2MB uss=155.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.0GB uss=12.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=11.1GB pss=11.0GB uss=11.0GB shared=48.1MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=280.8MB pss=207.7MB uss=155.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=281.7MB pss=208.3MB uss=155.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.1GB uss=12.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=280.9MB pss=207.9MB uss=155.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=281.8MB pss=208.4MB uss=155.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.1GB uss=12.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=281.0MB pss=207.9MB uss=155.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=13.1GB uss=12.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=281.1MB pss=208.0MB uss=155.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=13.1GB uss=12.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=281.2MB pss=208.2MB uss=155.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=281.9MB pss=208.5MB uss=155.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.1GB uss=12.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=11.2GB pss=11.1GB uss=11.1GB shared=48.1MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=281.3MB pss=208.2MB uss=155.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=13.2GB uss=13.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=281.8MB pss=208.7MB uss=156.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=13.2GB uss=13.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=281.9MB pss=208.8MB uss=156.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=13.2GB uss=13.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=282.0MB pss=208.9MB uss=156.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=282.0MB pss=208.6MB uss=155.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.2GB uss=13.0GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=11.3GB pss=11.2GB uss=11.2GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=13.3GB uss=13.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=282.1MB pss=209.0MB uss=156.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=282.1MB pss=208.7MB uss=156.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.3GB uss=13.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=282.2MB pss=208.8MB uss=156.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.3GB uss=13.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=282.3MB pss=208.9MB uss=156.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.3GB uss=13.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=282.2MB pss=209.1MB uss=156.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=283.3MB pss=209.9MB uss=157.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.3GB uss=13.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=282.3MB pss=209.2MB uss=156.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=13.3GB uss=13.1GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=11.4GB pss=11.3GB uss=11.3GB shared=48.1MB | |
| MEMORY: total (7 procs): pss=13.4GB uss=13.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=282.5MB pss=209.4MB uss=156.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=283.4MB pss=210.0MB uss=157.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.4GB uss=13.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=282.6MB pss=209.5MB uss=157.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=13.4GB uss=13.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=282.7MB pss=209.6MB uss=157.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=13.4GB uss=13.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=282.8MB pss=209.7MB uss=157.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=13.4GB uss=13.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=283.3MB pss=210.3MB uss=157.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=283.5MB pss=210.1MB uss=157.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.4GB uss=13.2GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=11.5GB pss=11.4GB uss=11.4GB shared=48.1MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=283.4MB pss=210.3MB uss=157.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=13.5GB uss=13.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=283.9MB pss=210.8MB uss=158.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=13.5GB uss=13.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=283.6MB pss=210.2MB uss=157.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.5GB uss=13.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=284.1MB pss=210.7MB uss=158.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.5GB uss=13.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=284.0MB pss=210.9MB uss=158.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=13.5GB uss=13.3GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=11.6GB pss=11.5GB uss=11.5GB shared=48.1MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=284.2MB pss=210.8MB uss=158.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.6GB uss=13.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=284.1MB pss=211.0MB uss=158.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=13.6GB uss=13.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=284.6MB pss=211.6MB uss=159.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=284.3MB pss=210.9MB uss=158.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.6GB uss=13.4GB | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=6.5GB, std_dev=186.9MB, min=459.6MB, max=6.6GB, num_seqs=4163, avg_data_len=1) | |
| train epoch 63, finished after 4163 steps, 0:31:57 elapsed (98.7% computing time) | |
| Save model under /u/maximilian.kannen/setups/20230406_feat/work/i6_core/returnn/training/ReturnnTrainingJob.wQIA3Rc1nLak/output/models/epoch.063 | |
| MEMORY: main proc python3(1781720) increased RSS: rss=11.8GB pss=11.8GB uss=11.7GB shared=48.2MB | |
| MEMORY: total (7 procs): pss=13.8GB uss=13.7GB | |
| epoch 63 score: 1.2383412008283545 error: None elapsed: 0:31:57 | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=6.6GB, std_dev=0.0B, min=6.6GB, max=6.6GB, num_seqs=31, avg_data_len=1) | |
| epoch 63 'dev' eval, finished after 31 steps, 0:00:07 elapsed (84.1% computing time) | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=284.7MB pss=211.6MB uss=159.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=284.8MB pss=211.5MB uss=158.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.8GB uss=13.7GB | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=6.6GB, std_dev=0.0B, min=6.6GB, max=6.6GB, num_seqs=29, avg_data_len=1) | |
| epoch 63 'devtrain' eval, finished after 29 steps, 0:00:03 elapsed (67.9% computing time) | |
| dev: score 1.6145227117316943 error None devtrain: score 1.6113956972862127 error None | |
| We have stored models for epochs [4, 5, 6, ..., 61, 62, 63] and keep epochs [4, 5, 6, 7, 8, 9, 10, 11, 59, 60, 61, 62, 63]. | |
| We will delete the models of epochs [58]. | |
| Deleted 335.5MB. | |
| start epoch 64 with learning rate 0.0001486125 ... | |
| [2023-10-31 00:43:31,222] INFO: [32mRun time: 0:38:51 CPU: 0.20% RSS: 14.77GB VMS: 50.37GB[0m | |
| [2023-10-31 00:54:52,937] INFO: [32mRun time: 0:50:13 CPU: 0.20% RSS: 16.26GB VMS: 52.40GB[0m | |
| MEMORY: sub proc MPD worker(1781782) increased RSS: rss=573.4MB pss=554.5MB uss=550.9MB shared=22.4MB | |
| MEMORY: sub proc MPD worker(1781783) increased RSS: rss=573.5MB pss=554.5MB uss=551.0MB shared=22.5MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=284.9MB pss=211.5MB uss=158.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.8GB uss=13.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=284.8MB pss=211.7MB uss=159.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=13.8GB uss=13.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=284.9MB pss=211.8MB uss=159.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=285.0MB pss=211.6MB uss=158.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.8GB uss=13.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=285.4MB pss=212.3MB uss=159.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=13.8GB uss=13.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=285.5MB pss=212.4MB uss=159.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=285.1MB pss=211.7MB uss=158.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.8GB uss=13.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=285.2MB pss=211.8MB uss=159.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.8GB uss=13.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=285.6MB pss=212.5MB uss=160.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=285.3MB pss=211.9MB uss=159.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.8GB uss=13.7GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=11.9GB pss=11.8GB uss=11.8GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=13.9GB uss=13.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=285.7MB pss=212.6MB uss=160.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=13.9GB uss=13.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=285.8MB pss=212.7MB uss=160.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=285.4MB pss=212.0MB uss=159.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=13.9GB uss=13.7GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=12.0GB pss=11.9GB uss=11.9GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=14.0GB uss=13.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=285.9MB pss=212.8MB uss=160.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=285.5MB pss=212.1MB uss=159.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.0GB uss=13.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=285.6MB pss=212.2MB uss=159.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.0GB uss=13.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=286.0MB pss=212.9MB uss=160.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=285.7MB pss=212.3MB uss=159.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.0GB uss=13.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=285.8MB pss=212.4MB uss=159.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.0GB uss=13.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=286.1MB pss=213.0MB uss=160.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=285.9MB pss=212.5MB uss=159.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.0GB uss=13.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=12.1GB pss=12.0GB uss=12.0GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=14.1GB uss=13.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=286.2MB pss=213.1MB uss=160.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=286.0MB pss=212.6MB uss=159.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.1GB uss=13.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=286.3MB pss=213.3MB uss=160.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=286.1MB pss=212.7MB uss=159.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.1GB uss=13.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=286.4MB pss=213.3MB uss=160.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=286.6MB pss=213.2MB uss=160.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.1GB uss=13.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=286.7MB pss=213.3MB uss=160.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.1GB uss=13.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=286.6MB pss=213.5MB uss=161.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=286.8MB pss=213.4MB uss=160.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.1GB uss=13.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=287.1MB pss=214.1MB uss=161.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=14.1GB uss=13.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=12.2GB pss=12.1GB uss=12.1GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=287.2MB pss=214.1MB uss=161.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=14.2GB uss=14.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=287.7MB pss=214.6MB uss=162.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=14.2GB uss=14.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=286.9MB pss=213.5MB uss=160.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.2GB uss=14.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=287.8MB pss=214.7MB uss=162.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=14.2GB uss=14.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=287.9MB pss=214.8MB uss=162.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=287.0MB pss=213.6MB uss=160.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.2GB uss=14.0GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=12.3GB pss=12.2GB uss=12.2GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=14.3GB uss=14.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=287.1MB pss=213.7MB uss=160.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.3GB uss=14.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=288.0MB pss=214.9MB uss=162.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=287.2MB pss=213.8MB uss=161.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.3GB uss=14.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=288.1MB pss=215.0MB uss=162.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=287.3MB pss=213.9MB uss=161.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.3GB uss=14.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=288.2MB pss=215.1MB uss=162.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=14.3GB uss=14.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=287.4MB pss=214.0MB uss=161.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.3GB uss=14.1GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=12.4GB pss=12.3GB uss=12.3GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=288.3MB pss=215.2MB uss=162.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=287.5MB pss=214.1MB uss=161.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.4GB uss=14.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=287.6MB pss=214.2MB uss=161.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.4GB uss=14.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=288.1MB pss=214.7MB uss=161.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.4GB uss=14.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=288.4MB pss=215.3MB uss=162.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=288.5MB pss=215.2MB uss=162.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.4GB uss=14.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=288.6MB pss=215.2MB uss=162.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.4GB uss=14.2GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=12.5GB pss=12.4GB uss=12.4GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=288.5MB pss=215.4MB uss=162.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=14.5GB uss=14.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=288.6MB pss=215.5MB uss=163.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=288.7MB pss=215.3MB uss=162.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.5GB uss=14.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=288.7MB pss=215.6MB uss=163.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=14.5GB uss=14.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=288.8MB pss=215.8MB uss=163.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=288.8MB pss=215.5MB uss=162.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.5GB uss=14.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=288.9MB pss=215.8MB uss=163.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=289.3MB pss=216.0MB uss=163.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.5GB uss=14.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=289.4MB pss=216.4MB uss=163.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=14.5GB uss=14.3GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=12.6GB pss=12.5GB uss=12.5GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=14.6GB uss=14.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=289.5MB pss=216.4MB uss=163.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=289.4MB pss=216.0MB uss=163.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.6GB uss=14.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=290.0MB pss=216.9MB uss=164.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=14.6GB uss=14.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=290.1MB pss=217.0MB uss=164.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=289.5MB pss=216.1MB uss=163.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.6GB uss=14.4GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=12.7GB pss=12.6GB uss=12.6GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=14.7GB uss=14.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=290.2MB pss=217.1MB uss=164.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=289.6MB pss=216.2MB uss=163.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.7GB uss=14.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=290.1MB pss=216.7MB uss=163.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.7GB uss=14.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=290.7MB pss=217.6MB uss=165.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=14.7GB uss=14.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=290.8MB pss=217.7MB uss=165.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=14.7GB uss=14.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=290.2MB pss=216.8MB uss=164.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.7GB uss=14.5GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=12.8GB pss=12.7GB uss=12.7GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=14.8GB uss=14.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=290.9MB pss=217.8MB uss=165.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=14.8GB uss=14.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=291.0MB pss=217.9MB uss=165.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=290.3MB pss=216.9MB uss=164.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.8GB uss=14.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=290.4MB pss=217.0MB uss=164.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.8GB uss=14.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=291.1MB pss=218.0MB uss=165.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=290.8MB pss=217.5MB uss=164.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.8GB uss=14.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=291.2MB pss=218.1MB uss=165.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=290.9MB pss=217.5MB uss=164.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.8GB uss=14.6GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=12.9GB pss=12.8GB uss=12.8GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=14.9GB uss=14.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=291.3MB pss=218.2MB uss=165.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=291.0MB pss=217.6MB uss=164.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.9GB uss=14.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=291.4MB pss=218.3MB uss=165.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=291.1MB pss=217.7MB uss=164.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.9GB uss=14.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=291.5MB pss=218.4MB uss=165.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=291.2MB pss=217.8MB uss=165.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=14.9GB uss=14.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=13.0GB pss=12.9GB uss=12.9GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=15.0GB uss=14.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=291.6MB pss=218.5MB uss=166.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=15.0GB uss=14.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=291.3MB pss=217.9MB uss=165.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.0GB uss=14.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=291.4MB pss=218.0MB uss=165.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.0GB uss=14.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=291.8MB pss=218.7MB uss=166.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=15.0GB uss=14.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=291.9MB pss=218.8MB uss=166.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=15.0GB uss=14.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=13.1GB pss=13.0GB uss=13.0GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=293.0MB pss=219.9MB uss=167.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=291.5MB pss=218.1MB uss=165.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.1GB uss=15.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=291.6MB pss=218.2MB uss=165.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.1GB uss=15.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=293.1MB pss=220.0MB uss=167.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=15.1GB uss=15.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=291.7MB pss=218.3MB uss=165.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.1GB uss=15.0GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=13.2GB pss=13.1GB uss=13.1GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=293.2MB pss=220.1MB uss=167.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=291.8MB pss=218.4MB uss=165.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.2GB uss=15.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=291.9MB pss=218.5MB uss=165.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.2GB uss=15.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=293.3MB pss=220.2MB uss=167.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=15.2GB uss=15.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=292.0MB pss=218.6MB uss=165.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.2GB uss=15.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=293.9MB pss=220.8MB uss=168.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=292.6MB pss=219.2MB uss=166.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.2GB uss=15.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=292.7MB pss=219.3MB uss=166.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.2GB uss=15.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=293.1MB pss=219.7MB uss=167.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.2GB uss=15.1GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=13.3GB pss=13.2GB uss=13.2GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=15.3GB uss=15.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=294.0MB pss=220.9MB uss=168.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=293.2MB pss=219.8MB uss=167.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.3GB uss=15.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=294.1MB pss=221.0MB uss=168.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=293.3MB pss=219.9MB uss=167.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.3GB uss=15.2GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=13.4GB pss=13.3GB uss=13.3GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=294.2MB pss=221.1MB uss=168.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=15.4GB uss=15.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=293.5MB pss=220.1MB uss=167.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.4GB uss=15.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=294.3MB pss=221.2MB uss=168.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=15.4GB uss=15.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=293.6MB pss=220.2MB uss=167.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.4GB uss=15.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=294.1MB pss=220.7MB uss=167.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.4GB uss=15.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=294.4MB pss=221.3MB uss=168.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=15.4GB uss=15.3GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=13.5GB pss=13.4GB uss=13.4GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=15.5GB uss=15.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=294.5MB pss=221.4MB uss=168.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=294.2MB pss=220.8MB uss=168.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.5GB uss=15.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=294.6MB pss=221.5MB uss=169.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=15.5GB uss=15.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=294.7MB pss=221.6MB uss=169.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=15.5GB uss=15.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=294.8MB pss=221.7MB uss=169.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=15.5GB uss=15.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=294.3MB pss=220.9MB uss=168.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.5GB uss=15.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=294.9MB pss=221.9MB uss=169.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=15.5GB uss=15.4GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=13.6GB pss=13.5GB uss=13.5GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=295.5MB pss=222.4MB uss=169.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=294.4MB pss=221.0MB uss=168.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.6GB uss=15.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=295.6MB pss=222.5MB uss=170.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=15.6GB uss=15.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=294.5MB pss=221.1MB uss=168.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.6GB uss=15.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=295.7MB pss=222.6MB uss=170.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=294.6MB pss=221.3MB uss=168.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.6GB uss=15.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=294.7MB pss=221.3MB uss=168.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.6GB uss=15.5GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=13.7GB pss=13.6GB uss=13.6GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=295.8MB pss=222.7MB uss=170.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=294.8MB pss=221.4MB uss=168.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.7GB uss=15.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=295.9MB pss=222.8MB uss=170.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=15.7GB uss=15.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=294.9MB pss=221.5MB uss=168.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.7GB uss=15.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=296.0MB pss=222.9MB uss=170.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=15.7GB uss=15.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=295.0MB pss=221.6MB uss=168.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.7GB uss=15.6GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=13.8GB pss=13.7GB uss=13.7GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=296.1MB pss=223.1MB uss=170.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=15.8GB uss=15.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=296.3MB pss=223.2MB uss=170.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=295.1MB pss=221.7MB uss=168.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.8GB uss=15.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=296.4MB pss=223.3MB uss=170.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=295.6MB pss=222.2MB uss=169.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.8GB uss=15.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=297.0MB pss=223.9MB uss=171.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=295.7MB pss=222.3MB uss=169.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.8GB uss=15.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=297.5MB pss=224.4MB uss=171.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=15.8GB uss=15.7GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=13.9GB pss=13.8GB uss=13.8GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=297.6MB pss=224.5MB uss=172.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=295.8MB pss=222.4MB uss=169.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.9GB uss=15.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=297.7MB pss=224.6MB uss=172.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=295.9MB pss=222.5MB uss=169.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.9GB uss=15.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=297.8MB pss=224.7MB uss=172.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=296.0MB pss=222.6MB uss=169.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=15.9GB uss=15.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=14.0GB pss=13.9GB uss=13.9GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=16.0GB uss=15.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=297.9MB pss=224.9MB uss=172.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=296.2MB pss=222.8MB uss=170.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.0GB uss=15.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=298.0MB pss=224.9MB uss=172.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=16.0GB uss=15.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=296.3MB pss=222.9MB uss=170.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.0GB uss=15.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=296.4MB pss=223.0MB uss=170.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.0GB uss=15.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=298.1MB pss=225.0MB uss=172.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=296.5MB pss=223.1MB uss=170.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.0GB uss=15.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=297.0MB pss=223.6MB uss=170.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.0GB uss=15.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=14.1GB pss=14.0GB uss=14.0GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=297.1MB pss=223.7MB uss=170.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.1GB uss=16.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=298.2MB pss=225.1MB uss=172.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=297.2MB pss=223.8MB uss=171.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.1GB uss=16.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=297.3MB pss=223.9MB uss=171.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.1GB uss=16.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=298.3MB pss=225.2MB uss=172.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=16.1GB uss=16.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=298.4MB pss=225.3MB uss=172.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=297.4MB pss=224.0MB uss=171.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.1GB uss=16.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=298.5MB pss=225.4MB uss=173.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=297.5MB pss=224.1MB uss=171.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.1GB uss=16.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=298.6MB pss=225.5MB uss=173.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=16.1GB uss=16.0GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=14.2GB pss=14.1GB uss=14.1GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=16.2GB uss=16.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=299.3MB pss=226.2MB uss=173.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=297.6MB pss=224.2MB uss=171.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.2GB uss=16.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=299.4MB pss=226.3MB uss=173.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=16.2GB uss=16.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=297.7MB pss=224.3MB uss=171.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.2GB uss=16.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=299.5MB pss=226.4MB uss=173.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=16.2GB uss=16.1GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=14.3GB pss=14.2GB uss=14.2GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=16.3GB uss=16.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=299.6MB pss=226.5MB uss=174.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=297.9MB pss=224.5MB uss=171.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.3GB uss=16.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=298.0MB pss=224.6MB uss=171.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.3GB uss=16.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=299.7MB pss=226.6MB uss=174.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=298.5MB pss=225.1MB uss=172.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.3GB uss=16.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=299.8MB pss=226.7MB uss=174.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=299.0MB pss=225.7MB uss=172.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.3GB uss=16.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=299.9MB pss=226.8MB uss=174.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=299.1MB pss=225.7MB uss=172.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.3GB uss=16.2GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=14.4GB pss=14.3GB uss=14.3GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=300.1MB pss=227.0MB uss=174.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=16.4GB uss=16.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=300.2MB pss=227.1MB uss=174.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=299.2MB pss=225.8MB uss=173.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.4GB uss=16.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=301.3MB pss=228.2MB uss=175.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=16.4GB uss=16.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=299.3MB pss=225.9MB uss=173.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.4GB uss=16.3GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=14.5GB pss=14.4GB uss=14.4GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=301.4MB pss=228.3MB uss=175.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=299.9MB pss=226.5MB uss=173.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.5GB uss=16.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=301.5MB pss=228.4MB uss=175.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=16.5GB uss=16.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=300.0MB pss=226.6MB uss=173.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.5GB uss=16.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=301.6MB pss=228.5MB uss=176.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=16.5GB uss=16.4GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=14.6GB pss=14.5GB uss=14.5GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=301.7MB pss=228.7MB uss=176.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=300.1MB pss=226.7MB uss=173.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.6GB uss=16.5GB | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=6.6GB, std_dev=0.0B, min=6.6GB, max=6.6GB, num_seqs=4175, avg_data_len=1) | |
| train epoch 64, finished after 4175 steps, 0:22:29 elapsed (99.5% computing time) | |
| Save model under /u/maximilian.kannen/setups/20230406_feat/work/i6_core/returnn/training/ReturnnTrainingJob.wQIA3Rc1nLak/output/models/epoch.064 | |
| MEMORY: main proc python3(1781720) increased RSS: rss=14.7GB pss=14.7GB uss=14.7GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=301.8MB pss=228.7MB uss=176.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=16.8GB uss=16.6GB | |
| epoch 64 score: 1.2373794239855929 error: None elapsed: 0:22:29 | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=6.6GB, std_dev=0.0B, min=6.6GB, max=6.6GB, num_seqs=31, avg_data_len=1) | |
| epoch 64 'dev' eval, finished after 31 steps, 0:00:03 elapsed (62.2% computing time) | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=6.6GB, std_dev=0.0B, min=6.6GB, max=6.6GB, num_seqs=29, avg_data_len=1) | |
| epoch 64 'devtrain' eval, finished after 29 steps, 0:00:03 elapsed (61.3% computing time) | |
| dev: score 1.4426778692631834 error None devtrain: score 1.4329270127424272 error None | |
| We have stored models for epochs [4, 5, 6, ..., 62, 63, 64] and keep epochs [4, 5, 6, 7, 8, 9, 10, 11, 60, 61, 62, 63, 64]. | |
| We will delete the models of epochs [59]. | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=301.9MB pss=228.8MB uss=176.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=300.3MB pss=227.0MB uss=174.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.8GB uss=16.6GB | |
| Deleted 335.5MB. | |
| start epoch 65 with learning rate 0.0001507611111111111 ... | |
| [2023-10-31 01:08:09,960] INFO: [32mRun time: 1:03:30 CPU: 0.20% RSS: 17.89GB VMS: 54.56GB[0m | |
| MEMORY: sub proc MPD worker(1781782) increased RSS: rss=573.6MB pss=554.7MB uss=551.2MB shared=22.4MB | |
| MEMORY: sub proc MPD worker(1781783) increased RSS: rss=573.7MB pss=554.8MB uss=551.3MB shared=22.5MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=300.4MB pss=227.0MB uss=174.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.8GB uss=16.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=300.5MB pss=227.1MB uss=174.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.8GB uss=16.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=302.0MB pss=228.9MB uss=176.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=300.6MB pss=227.2MB uss=174.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.8GB uss=16.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=302.1MB pss=229.0MB uss=176.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=16.8GB uss=16.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=302.2MB pss=229.1MB uss=176.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=16.8GB uss=16.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=302.3MB pss=229.2MB uss=176.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=300.7MB pss=227.3MB uss=174.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.8GB uss=16.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=302.4MB pss=229.3MB uss=176.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=16.8GB uss=16.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=302.6MB pss=229.5MB uss=177.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=300.8MB pss=227.4MB uss=174.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.8GB uss=16.6GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=14.8GB pss=14.7GB uss=14.7GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=303.1MB pss=230.0MB uss=177.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=16.8GB uss=16.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=303.2MB pss=230.1MB uss=177.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=16.8GB uss=16.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=303.3MB pss=230.2MB uss=177.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=300.9MB pss=227.6MB uss=174.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.8GB uss=16.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=301.5MB pss=228.1MB uss=175.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.8GB uss=16.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=301.6MB pss=228.2MB uss=175.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.8GB uss=16.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=303.4MB pss=230.3MB uss=177.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=301.7MB pss=228.3MB uss=175.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.8GB uss=16.7GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=14.9GB pss=14.8GB uss=14.8GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=16.9GB uss=16.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=303.5MB pss=230.4MB uss=177.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=16.9GB uss=16.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=303.6MB pss=230.5MB uss=178.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=301.8MB pss=228.4MB uss=175.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.9GB uss=16.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=303.7MB pss=230.6MB uss=178.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=16.9GB uss=16.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=303.9MB pss=230.8MB uss=178.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=301.9MB pss=228.5MB uss=175.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=16.9GB uss=16.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=15.0GB pss=14.9GB uss=14.9GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=304.0MB pss=230.9MB uss=178.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=302.0MB pss=228.6MB uss=175.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.0GB uss=16.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=304.6MB pss=231.6MB uss=179.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=302.1MB pss=228.8MB uss=176.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.0GB uss=16.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=304.7MB pss=231.6MB uss=179.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=302.2MB pss=228.8MB uss=176.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.0GB uss=16.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=302.3MB pss=228.9MB uss=176.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.0GB uss=16.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=304.8MB pss=231.7MB uss=179.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.0GB uss=16.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=304.9MB pss=231.8MB uss=179.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=302.4MB pss=229.0MB uss=176.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.0GB uss=16.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=305.0MB pss=231.9MB uss=179.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=302.5MB pss=229.1MB uss=176.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.0GB uss=16.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=303.0MB pss=229.6MB uss=176.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.0GB uss=16.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=15.1GB pss=15.0GB uss=15.0GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=305.1MB pss=232.0MB uss=179.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.1GB uss=17.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=305.3MB pss=232.2MB uss=179.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=303.1MB pss=229.8MB uss=177.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.1GB uss=17.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=305.4MB pss=232.3MB uss=179.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=303.2MB pss=229.8MB uss=177.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.1GB uss=17.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=305.9MB pss=232.9MB uss=180.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.1GB uss=17.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=306.0MB pss=232.9MB uss=180.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.1GB uss=17.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=303.3MB pss=229.9MB uss=177.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.1GB uss=17.0GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=15.2GB pss=15.1GB uss=15.1GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=17.2GB uss=17.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=306.1MB pss=233.0MB uss=180.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.2GB uss=17.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=303.4MB pss=230.0MB uss=177.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.2GB uss=17.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=306.2MB pss=233.1MB uss=180.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=303.6MB pss=230.2MB uss=177.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.2GB uss=17.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=306.3MB pss=233.2MB uss=180.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.2GB uss=17.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=303.7MB pss=230.3MB uss=177.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.2GB uss=17.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=306.4MB pss=233.3MB uss=180.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.2GB uss=17.1GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=15.3GB pss=15.2GB uss=15.2GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=17.3GB uss=17.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=306.5MB pss=233.4MB uss=180.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=303.9MB pss=230.5MB uss=177.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.3GB uss=17.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=304.0MB pss=230.6MB uss=177.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.3GB uss=17.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=304.5MB pss=231.1MB uss=178.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.3GB uss=17.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=306.6MB pss=233.5MB uss=181.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=305.0MB pss=231.6MB uss=178.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.3GB uss=17.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=305.1MB pss=231.7MB uss=178.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.3GB uss=17.2GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=15.4GB pss=15.3GB uss=15.3GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=306.7MB pss=233.6MB uss=181.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.4GB uss=17.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=306.8MB pss=233.7MB uss=181.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.4GB uss=17.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=306.9MB pss=233.8MB uss=181.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.4GB uss=17.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=307.0MB pss=233.9MB uss=181.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=305.2MB pss=231.8MB uss=179.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.4GB uss=17.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=307.1MB pss=234.0MB uss=181.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=305.3MB pss=231.9MB uss=179.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.4GB uss=17.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=307.6MB pss=234.5MB uss=182.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.4GB uss=17.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=307.7MB pss=234.6MB uss=182.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.4GB uss=17.3GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=15.5GB pss=15.4GB uss=15.4GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=305.9MB pss=232.5MB uss=179.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.5GB uss=17.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=307.8MB pss=234.7MB uss=182.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.5GB uss=17.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=307.9MB pss=234.8MB uss=182.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=306.0MB pss=232.6MB uss=179.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.5GB uss=17.4GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=15.6GB pss=15.5GB uss=15.5GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=308.0MB pss=234.9MB uss=182.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=306.1MB pss=232.7MB uss=179.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.6GB uss=17.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=308.1MB pss=235.0MB uss=182.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.6GB uss=17.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=306.2MB pss=232.8MB uss=180.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.6GB uss=17.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=308.2MB pss=235.1MB uss=182.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.6GB uss=17.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=308.4MB pss=235.3MB uss=182.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=306.3MB pss=233.0MB uss=180.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.6GB uss=17.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=308.5MB pss=235.4MB uss=182.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=306.4MB pss=233.0MB uss=180.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.6GB uss=17.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=309.1MB pss=236.0MB uss=183.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.6GB uss=17.5GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=15.7GB pss=15.6GB uss=15.6GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=309.2MB pss=236.1MB uss=183.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.7GB uss=17.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=306.5MB pss=233.1MB uss=180.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.7GB uss=17.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=309.3MB pss=236.3MB uss=183.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.7GB uss=17.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=309.5MB pss=236.4MB uss=183.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.7GB uss=17.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=306.6MB pss=233.2MB uss=180.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.7GB uss=17.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=309.6MB pss=236.5MB uss=184.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.7GB uss=17.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=310.1MB pss=237.0MB uss=184.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.7GB uss=17.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=306.7MB pss=233.4MB uss=180.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.7GB uss=17.6GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=15.8GB pss=15.7GB uss=15.7GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=306.8MB pss=233.5MB uss=180.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.8GB uss=17.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=306.9MB pss=233.5MB uss=180.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.8GB uss=17.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=307.0MB pss=233.6MB uss=180.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.8GB uss=17.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=307.6MB pss=234.2MB uss=181.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.8GB uss=17.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=307.7MB pss=234.3MB uss=181.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.8GB uss=17.7GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=15.9GB pss=15.8GB uss=15.8GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=17.9GB uss=17.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=310.7MB pss=237.6MB uss=185.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=307.8MB pss=234.4MB uss=181.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.9GB uss=17.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=310.8MB pss=237.7MB uss=185.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.9GB uss=17.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=310.9MB pss=237.8MB uss=185.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=17.9GB uss=17.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=311.0MB pss=237.9MB uss=185.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=307.9MB pss=234.6MB uss=181.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.9GB uss=17.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=308.0MB pss=234.6MB uss=181.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=17.9GB uss=17.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=16.0GB pss=15.9GB uss=15.9GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=18.0GB uss=17.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=311.1MB pss=238.0MB uss=185.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=308.2MB pss=234.8MB uss=182.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.0GB uss=17.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=308.3MB pss=234.9MB uss=182.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.0GB uss=17.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=311.2MB pss=238.1MB uss=185.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=308.4MB pss=235.0MB uss=182.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.0GB uss=17.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=309.0MB pss=235.6MB uss=182.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.0GB uss=17.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=311.3MB pss=238.2MB uss=185.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=18.0GB uss=17.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=309.1MB pss=235.7MB uss=182.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.0GB uss=17.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=16.1GB pss=16.0GB uss=16.0GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=311.4MB pss=238.3MB uss=185.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=18.1GB uss=18.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=311.5MB pss=238.5MB uss=186.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=309.2MB pss=235.8MB uss=183.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.1GB uss=18.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=311.6MB pss=238.5MB uss=186.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=18.1GB uss=18.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=312.1MB pss=239.0MB uss=186.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=18.1GB uss=18.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=312.5MB pss=239.5MB uss=187.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=18.1GB uss=18.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=312.6MB pss=239.5MB uss=187.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=18.1GB uss=18.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=309.3MB pss=235.9MB uss=183.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.1GB uss=18.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=312.7MB pss=239.6MB uss=187.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=18.1GB uss=18.0GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=16.2GB pss=16.1GB uss=16.1GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=18.2GB uss=18.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=309.4MB pss=236.0MB uss=183.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.2GB uss=18.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=312.8MB pss=239.7MB uss=187.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=309.5MB pss=236.1MB uss=183.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.2GB uss=18.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=313.3MB pss=240.3MB uss=187.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=309.6MB pss=236.2MB uss=183.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.2GB uss=18.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=313.4MB pss=240.3MB uss=187.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=309.7MB pss=236.3MB uss=183.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.2GB uss=18.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=309.8MB pss=236.4MB uss=183.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.2GB uss=18.1GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=16.3GB pss=16.2GB uss=16.2GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=313.5MB pss=240.4MB uss=187.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=310.0MB pss=236.6MB uss=183.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.3GB uss=18.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=310.5MB pss=237.1MB uss=184.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.3GB uss=18.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=313.6MB pss=240.5MB uss=188.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=310.6MB pss=237.2MB uss=184.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.3GB uss=18.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=313.7MB pss=240.6MB uss=188.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=18.3GB uss=18.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=313.8MB pss=240.7MB uss=188.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=310.7MB pss=237.3MB uss=184.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.3GB uss=18.2GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=16.4GB pss=16.3GB uss=16.3GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=313.9MB pss=240.8MB uss=188.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=18.4GB uss=18.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=310.8MB pss=237.4MB uss=184.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.4GB uss=18.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=314.0MB pss=240.9MB uss=188.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=310.9MB pss=237.5MB uss=184.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.4GB uss=18.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=314.1MB pss=241.0MB uss=188.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=311.1MB pss=237.7MB uss=184.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.4GB uss=18.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=314.2MB pss=241.1MB uss=188.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=311.2MB pss=237.8MB uss=185.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.4GB uss=18.3GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=16.5GB pss=16.4GB uss=16.4GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=18.5GB uss=18.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=314.3MB pss=241.3MB uss=188.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=311.3MB pss=237.9MB uss=185.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.5GB uss=18.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=314.5MB pss=241.4MB uss=188.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=311.4MB pss=238.0MB uss=185.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.5GB uss=18.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=311.5MB pss=238.1MB uss=185.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.5GB uss=18.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=312.0MB pss=238.6MB uss=185.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.5GB uss=18.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=314.6MB pss=241.5MB uss=189.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=18.5GB uss=18.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=315.6MB pss=242.5MB uss=190.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=312.1MB pss=238.7MB uss=185.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.5GB uss=18.4GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=16.6GB pss=16.5GB uss=16.5GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=315.7MB pss=242.6MB uss=190.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=18.6GB uss=18.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=312.2MB pss=238.8MB uss=186.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.6GB uss=18.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=315.8MB pss=242.7MB uss=190.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=18.6GB uss=18.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=312.3MB pss=238.9MB uss=186.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.6GB uss=18.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=315.9MB pss=242.8MB uss=190.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=18.6GB uss=18.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=316.0MB pss=242.9MB uss=190.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=18.6GB uss=18.5GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=16.7GB pss=16.6GB uss=16.6GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=312.4MB pss=239.0MB uss=186.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.7GB uss=18.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=316.6MB pss=243.5MB uss=191.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=312.5MB pss=239.2MB uss=186.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.7GB uss=18.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=312.6MB pss=239.2MB uss=186.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.7GB uss=18.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=312.7MB pss=239.3MB uss=186.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.7GB uss=18.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=316.7MB pss=243.6MB uss=191.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=312.8MB pss=239.4MB uss=186.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.7GB uss=18.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=312.9MB pss=239.5MB uss=186.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.7GB uss=18.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=313.4MB pss=240.0MB uss=187.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.7GB uss=18.6GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=16.8GB pss=16.7GB uss=16.7GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=18.8GB uss=18.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=316.8MB pss=243.8MB uss=191.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=313.9MB pss=240.6MB uss=187.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.8GB uss=18.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=316.9MB pss=243.8MB uss=191.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=314.0MB pss=240.6MB uss=187.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.8GB uss=18.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=317.0MB pss=243.9MB uss=191.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=18.8GB uss=18.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=314.1MB pss=240.7MB uss=187.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.8GB uss=18.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=317.1MB pss=244.0MB uss=191.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=18.8GB uss=18.7GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=16.9GB pss=16.8GB uss=16.8GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=314.2MB pss=240.8MB uss=188.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.9GB uss=18.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=314.3MB pss=240.9MB uss=188.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.9GB uss=18.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=317.2MB pss=244.1MB uss=191.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=18.9GB uss=18.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=317.3MB pss=244.3MB uss=191.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=314.8MB pss=241.5MB uss=188.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.9GB uss=18.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=317.4MB pss=244.3MB uss=191.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=18.9GB uss=18.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=314.9MB pss=241.5MB uss=188.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=18.9GB uss=18.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=317.5MB pss=244.4MB uss=191.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=18.9GB uss=18.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=318.1MB pss=245.0MB uss=192.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=18.9GB uss=18.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=318.2MB pss=245.1MB uss=192.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=18.9GB uss=18.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=17.0GB pss=16.9GB uss=16.9GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=315.0MB pss=241.6MB uss=188.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.0GB uss=18.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=318.3MB pss=245.2MB uss=192.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.0GB uss=18.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=315.1MB pss=241.7MB uss=188.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.0GB uss=18.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=318.4MB pss=245.3MB uss=192.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.0GB uss=18.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=315.2MB pss=241.8MB uss=189.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.0GB uss=18.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=315.3MB pss=241.9MB uss=189.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.0GB uss=18.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=318.5MB pss=245.4MB uss=192.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.0GB uss=18.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=17.1GB pss=17.0GB uss=17.0GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=19.1GB uss=19.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=318.6MB pss=245.5MB uss=193.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.1GB uss=19.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=318.7MB pss=245.6MB uss=193.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=315.4MB pss=242.0MB uss=189.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.1GB uss=19.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=318.8MB pss=245.8MB uss=193.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.1GB uss=19.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=319.0MB pss=245.9MB uss=193.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=315.5MB pss=242.1MB uss=189.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.1GB uss=19.0GB | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=6.6GB, std_dev=1.0MB, min=6.6GB, max=6.6GB, num_seqs=4190, avg_data_len=1) | |
| train epoch 65, finished after 4190 steps, 0:21:26 elapsed (99.5% computing time) | |
| Save model under /u/maximilian.kannen/setups/20230406_feat/work/i6_core/returnn/training/ReturnnTrainingJob.wQIA3Rc1nLak/output/models/epoch.065 | |
| [2023-10-31 01:21:52,029] INFO: [32mRun time: 1:17:12 CPU: 0.20% RSS: 19.74GB VMS: 57.09GB[0m | |
| MEMORY: main proc python3(1781720) increased RSS: rss=17.4GB pss=17.3GB uss=17.3GB shared=49.3MB | |
| epoch 65 score: 1.2422427488333752 error: None elapsed: 0:21:26 | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=6.6GB, std_dev=0.0B, min=6.6GB, max=6.6GB, num_seqs=31, avg_data_len=1) | |
| epoch 65 'dev' eval, finished after 31 steps, 0:00:03 elapsed (62.2% computing time) | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=319.6MB pss=246.5MB uss=194.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=315.6MB pss=242.2MB uss=189.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=6.6GB, std_dev=0.0B, min=6.6GB, max=6.6GB, num_seqs=29, avg_data_len=1) | |
| epoch 65 'devtrain' eval, finished after 29 steps, 0:00:03 elapsed (61.0% computing time) | |
| dev: score 1.3959701418159443 error None devtrain: score 1.3859798670430228 error None | |
| We have stored models for epochs [4, 5, 6, ..., 63, 64, 65] and keep epochs [4, 5, 6, 7, 8, 9, 10, 11, 61, 62, 63, 64, 65]. | |
| We will delete the models of epochs [60]. | |
| Deleted 335.5MB. | |
| start epoch 66 with learning rate 0.00015290972222222222 ... | |
| [2023-10-31 01:41:40,053] INFO: [32mRun time: 1:37:00 CPU: 0.40% RSS: 21.72GB VMS: 59.63GB[0m | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=319.7MB pss=246.6MB uss=194.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=315.7MB pss=242.3MB uss=189.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=315.8MB pss=242.4MB uss=189.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=319.8MB pss=246.7MB uss=194.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=319.9MB pss=246.8MB uss=194.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=315.9MB pss=242.5MB uss=189.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=316.5MB pss=243.1MB uss=190.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=320.0MB pss=246.9MB uss=194.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=316.6MB pss=243.2MB uss=190.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=316.7MB pss=243.3MB uss=190.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=320.1MB pss=247.0MB uss=194.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=320.2MB pss=247.1MB uss=194.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=320.3MB pss=247.2MB uss=194.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=316.8MB pss=243.4MB uss=190.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=320.4MB pss=247.4MB uss=194.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=320.5MB pss=247.4MB uss=194.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=316.9MB pss=243.5MB uss=190.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=320.6MB pss=247.5MB uss=195.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=321.6MB pss=248.5MB uss=196.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=317.1MB pss=243.7MB uss=190.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=317.2MB pss=243.8MB uss=191.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=321.7MB pss=248.6MB uss=196.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=317.3MB pss=243.9MB uss=191.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=317.4MB pss=244.0MB uss=191.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=317.9MB pss=244.6MB uss=191.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=321.8MB pss=248.7MB uss=196.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=318.0MB pss=244.6MB uss=191.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=321.9MB pss=248.8MB uss=196.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=318.2MB pss=244.8MB uss=192.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.4GB uss=19.3GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=17.5GB pss=17.4GB uss=17.4GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=322.0MB pss=248.9MB uss=196.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.5GB uss=19.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=318.3MB pss=244.9MB uss=192.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.5GB uss=19.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=322.5MB pss=249.5MB uss=197.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.5GB uss=19.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=322.6MB pss=249.5MB uss=197.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=318.4MB pss=245.0MB uss=192.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.5GB uss=19.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=322.7MB pss=249.6MB uss=197.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.5GB uss=19.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=318.5MB pss=245.1MB uss=192.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.5GB uss=19.4GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=17.6GB pss=17.5GB uss=17.5GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=19.6GB uss=19.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=322.8MB pss=249.7MB uss=197.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=318.6MB pss=245.2MB uss=192.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.6GB uss=19.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=318.7MB pss=245.3MB uss=192.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.6GB uss=19.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=322.9MB pss=249.8MB uss=197.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=318.8MB pss=245.4MB uss=192.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.6GB uss=19.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=318.9MB pss=245.5MB uss=192.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.6GB uss=19.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=319.4MB pss=246.0MB uss=193.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.6GB uss=19.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=323.0MB pss=249.9MB uss=197.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=319.9MB pss=246.6MB uss=193.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.6GB uss=19.5GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=17.7GB pss=17.6GB uss=17.6GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=19.7GB uss=19.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=323.1MB pss=250.0MB uss=197.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=320.0MB pss=246.6MB uss=193.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.7GB uss=19.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=323.2MB pss=250.1MB uss=197.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.7GB uss=19.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=323.4MB pss=250.3MB uss=197.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.7GB uss=19.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=320.1MB pss=246.7MB uss=193.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.7GB uss=19.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=323.5MB pss=250.4MB uss=197.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.7GB uss=19.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=324.1MB pss=251.0MB uss=198.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=320.2MB pss=246.8MB uss=194.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.7GB uss=19.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=324.2MB pss=251.1MB uss=198.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.7GB uss=19.6GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=17.8GB pss=17.7GB uss=17.7GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=19.8GB uss=19.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=320.3MB pss=246.9MB uss=194.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.8GB uss=19.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=324.4MB pss=251.3MB uss=198.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=320.4MB pss=247.0MB uss=194.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.8GB uss=19.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=324.5MB pss=251.4MB uss=198.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.8GB uss=19.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=324.6MB pss=251.5MB uss=199.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=320.9MB pss=247.5MB uss=194.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.8GB uss=19.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=324.7MB pss=251.6MB uss=199.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.8GB uss=19.7GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=17.9GB pss=17.8GB uss=17.8GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=19.9GB uss=19.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=324.8MB pss=251.7MB uss=199.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.9GB uss=19.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=324.9MB pss=251.9MB uss=199.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.9GB uss=19.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=325.0MB pss=251.9MB uss=199.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=19.9GB uss=19.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=325.1MB pss=252.0MB uss=199.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=321.0MB pss=247.6MB uss=194.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=19.9GB uss=19.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=18.0GB pss=17.9GB uss=17.9GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=325.7MB pss=252.6MB uss=200.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=321.1MB pss=247.7MB uss=194.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.0GB uss=19.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=325.8MB pss=252.7MB uss=200.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=321.7MB pss=248.3MB uss=195.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.0GB uss=19.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=325.9MB pss=252.8MB uss=200.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=321.8MB pss=248.4MB uss=195.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.0GB uss=19.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=321.9MB pss=248.5MB uss=195.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.0GB uss=19.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=326.0MB pss=252.9MB uss=200.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=20.0GB uss=19.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=18.1GB pss=18.0GB uss=18.0GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=20.1GB uss=20.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=326.2MB pss=253.1MB uss=200.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=322.0MB pss=248.6MB uss=195.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.1GB uss=20.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=326.3MB pss=253.2MB uss=200.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=20.1GB uss=20.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=322.1MB pss=248.7MB uss=195.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.1GB uss=20.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=326.4MB pss=253.3MB uss=200.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=20.1GB uss=20.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=326.6MB pss=253.5MB uss=201.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=322.3MB pss=248.9MB uss=196.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.1GB uss=20.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=322.4MB pss=249.0MB uss=196.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.1GB uss=20.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=327.1MB pss=254.0MB uss=201.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=20.1GB uss=20.0GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=18.2GB pss=18.1GB uss=18.1GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=20.2GB uss=20.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=327.2MB pss=254.1MB uss=201.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=322.6MB pss=249.2MB uss=196.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.2GB uss=20.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=322.7MB pss=249.3MB uss=196.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.2GB uss=20.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=327.3MB pss=254.2MB uss=201.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=20.2GB uss=20.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=323.3MB pss=249.9MB uss=197.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.2GB uss=20.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=323.4MB pss=250.0MB uss=197.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.2GB uss=20.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=327.4MB pss=254.3MB uss=201.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=20.2GB uss=20.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=323.5MB pss=250.1MB uss=197.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.2GB uss=20.1GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=18.3GB pss=18.2GB uss=18.2GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=327.5MB pss=254.5MB uss=202.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=20.3GB uss=20.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=327.6MB pss=254.5MB uss=202.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=20.3GB uss=20.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=323.6MB pss=250.2MB uss=197.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.3GB uss=20.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=327.8MB pss=254.7MB uss=202.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=20.3GB uss=20.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=327.9MB pss=254.8MB uss=202.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=20.3GB uss=20.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=328.6MB pss=255.5MB uss=203.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=323.7MB pss=250.3MB uss=197.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.4GB uss=20.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=328.7MB pss=255.6MB uss=203.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=323.8MB pss=250.4MB uss=197.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.4GB uss=20.2GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=18.4GB pss=18.3GB uss=18.3GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=20.5GB uss=20.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=328.8MB pss=255.7MB uss=203.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=323.9MB pss=250.5MB uss=197.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.5GB uss=20.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=329.0MB pss=255.9MB uss=203.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=324.0MB pss=250.7MB uss=197.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.5GB uss=20.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=324.1MB pss=250.7MB uss=197.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.5GB uss=20.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=329.1MB pss=256.0MB uss=203.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=20.5GB uss=20.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=329.2MB pss=256.1MB uss=203.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=324.8MB pss=251.4MB uss=198.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.5GB uss=20.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=324.9MB pss=251.5MB uss=198.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.5GB uss=20.3GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=18.5GB pss=18.4GB uss=18.4GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=20.6GB uss=20.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=329.3MB pss=256.2MB uss=203.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=20.6GB uss=20.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=325.0MB pss=251.6MB uss=198.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.6GB uss=20.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=329.4MB pss=256.3MB uss=203.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=20.6GB uss=20.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=329.5MB pss=256.4MB uss=203.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=20.6GB uss=20.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=325.1MB pss=251.7MB uss=198.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.6GB uss=20.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=329.6MB pss=256.5MB uss=204.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=20.6GB uss=20.4GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=18.6GB pss=18.5GB uss=18.5GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=330.6MB pss=257.5MB uss=205.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=325.2MB pss=251.8MB uss=199.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.6GB uss=20.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=325.3MB pss=251.9MB uss=199.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.6GB uss=20.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=330.7MB pss=257.6MB uss=205.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=325.4MB pss=252.0MB uss=199.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.6GB uss=20.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=325.5MB pss=252.1MB uss=199.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.6GB uss=20.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=330.8MB pss=257.7MB uss=205.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=325.6MB pss=252.2MB uss=199.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.6GB uss=20.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=330.9MB pss=257.8MB uss=205.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=325.8MB pss=252.4MB uss=199.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.6GB uss=20.5GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=18.7GB pss=18.6GB uss=18.6GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=326.3MB pss=252.9MB uss=200.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.8GB uss=20.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=331.0MB pss=257.9MB uss=205.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=20.8GB uss=20.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=331.5MB pss=258.4MB uss=206.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=326.5MB pss=253.1MB uss=200.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.8GB uss=20.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=331.6MB pss=258.5MB uss=206.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=326.6MB pss=253.2MB uss=200.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.8GB uss=20.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=326.7MB pss=253.3MB uss=200.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.8GB uss=20.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=331.7MB pss=258.6MB uss=206.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=20.8GB uss=20.6GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=18.8GB pss=18.7GB uss=18.7GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=20.9GB uss=20.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=326.8MB pss=253.4MB uss=200.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.9GB uss=20.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=331.8MB pss=258.8MB uss=206.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=20.9GB uss=20.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=331.9MB pss=258.8MB uss=206.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=20.9GB uss=20.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=332.0MB pss=258.9MB uss=206.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=326.9MB pss=253.5MB uss=200.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.9GB uss=20.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=332.1MB pss=259.0MB uss=206.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=327.0MB pss=253.6MB uss=200.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.9GB uss=20.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=327.1MB pss=253.7MB uss=200.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=20.9GB uss=20.7GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=18.9GB pss=18.8GB uss=18.8GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=332.2MB pss=259.1MB uss=206.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=327.2MB pss=253.8MB uss=201.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.0GB uss=20.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=327.7MB pss=254.4MB uss=201.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.0GB uss=20.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=327.8MB pss=254.5MB uss=201.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.0GB uss=20.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=332.3MB pss=259.3MB uss=206.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=327.9MB pss=254.5MB uss=201.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.0GB uss=20.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=332.4MB pss=259.3MB uss=206.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=21.0GB uss=20.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=332.5MB pss=259.4MB uss=206.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=21.0GB uss=20.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=332.6MB pss=259.5MB uss=207.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=328.0MB pss=254.6MB uss=201.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.0GB uss=20.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=333.1MB pss=260.1MB uss=207.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=21.0GB uss=20.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=19.0GB pss=18.9GB uss=18.9GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=333.2MB pss=260.1MB uss=207.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=21.1GB uss=20.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=333.7MB pss=260.6MB uss=208.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=328.1MB pss=254.7MB uss=201.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.1GB uss=20.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=328.6MB pss=255.2MB uss=202.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.1GB uss=20.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=333.8MB pss=260.7MB uss=208.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=21.1GB uss=20.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=333.9MB pss=260.8MB uss=208.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=21.1GB uss=20.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=19.1GB pss=19.0GB uss=19.0GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=21.2GB uss=21.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=334.1MB pss=261.0MB uss=208.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=21.2GB uss=21.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=328.7MB pss=255.3MB uss=202.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.2GB uss=21.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=334.2MB pss=261.1MB uss=208.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=21.2GB uss=21.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=334.7MB pss=261.7MB uss=209.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=329.8MB pss=256.4MB uss=203.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.2GB uss=21.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=334.8MB pss=261.7MB uss=209.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=21.2GB uss=21.0GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=19.2GB pss=19.1GB uss=19.1GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=334.9MB pss=261.8MB uss=209.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=329.9MB pss=256.5MB uss=203.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.3GB uss=21.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=335.0MB pss=262.0MB uss=209.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=330.0MB pss=256.6MB uss=203.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.3GB uss=21.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=335.1MB pss=262.0MB uss=209.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=21.3GB uss=21.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=330.1MB pss=256.7MB uss=203.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.3GB uss=21.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=335.2MB pss=262.2MB uss=209.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=330.2MB pss=256.8MB uss=204.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.3GB uss=21.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=335.3MB pss=262.2MB uss=209.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=330.7MB pss=257.3MB uss=204.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.3GB uss=21.1GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=19.3GB pss=19.2GB uss=19.2GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=21.4GB uss=21.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=335.4MB pss=262.3MB uss=209.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=330.8MB pss=257.4MB uss=204.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.4GB uss=21.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=335.5MB pss=262.4MB uss=209.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=21.4GB uss=21.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=330.9MB pss=257.5MB uss=204.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.4GB uss=21.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=335.6MB pss=262.5MB uss=210.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=21.4GB uss=21.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=336.2MB pss=263.1MB uss=210.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=331.0MB pss=257.6MB uss=204.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.4GB uss=21.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=336.3MB pss=263.2MB uss=210.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=21.4GB uss=21.2GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=19.4GB pss=19.3GB uss=19.3GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=336.4MB pss=263.3MB uss=210.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=331.1MB pss=257.7MB uss=204.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.5GB uss=21.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=336.5MB pss=263.4MB uss=210.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=21.5GB uss=21.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=331.2MB pss=257.8MB uss=205.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.5GB uss=21.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=336.6MB pss=263.5MB uss=211.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=21.5GB uss=21.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=331.3MB pss=257.9MB uss=205.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.5GB uss=21.3GB | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=7.1GB, std_dev=478.8MB, min=6.6GB, max=7.5GB, num_seqs=4199, avg_data_len=1) | |
| train epoch 66, finished after 4199 steps, 0:20:58 elapsed (99.5% computing time) | |
| Save model under /u/maximilian.kannen/setups/20230406_feat/work/i6_core/returnn/training/ReturnnTrainingJob.wQIA3Rc1nLak/output/models/epoch.066 | |
| MEMORY: main proc python3(1781720) increased RSS: rss=19.6GB pss=19.6GB uss=19.6GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=21.7GB uss=21.6GB | |
| epoch 66 score: 1.239375407121415 error: None elapsed: 0:20:58 | |
| MEMORY: main proc python3(1781720) increased RSS: rss=19.7GB pss=19.6GB uss=19.6GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=21.8GB uss=21.7GB | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=7.5GB, std_dev=0.0B, min=7.5GB, max=7.5GB, num_seqs=31, avg_data_len=1) | |
| epoch 66 'dev' eval, finished after 31 steps, 0:00:03 elapsed (63.2% computing time) | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=7.5GB, std_dev=0.0B, min=7.5GB, max=7.5GB, num_seqs=29, avg_data_len=1) | |
| epoch 66 'devtrain' eval, finished after 29 steps, 0:00:03 elapsed (60.6% computing time) | |
| dev: score 1.381709769024575 error None devtrain: score 1.371463269147463 error None | |
| We have stored models for epochs [4, 5, 6, ..., 64, 65, 66] and keep epochs [4, 5, 6, 7, 8, 9, 10, 11, 62, 63, 64, 65, 66]. | |
| We will delete the models of epochs [61]. | |
| Deleted 335.5MB. | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=336.8MB pss=263.7MB uss=211.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=331.5MB pss=258.1MB uss=205.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.8GB uss=21.7GB | |
| start epoch 67 with learning rate 0.00015505833333333334 ... | |
| [2023-10-31 02:01:48,207] INFO: [32mRun time: 1:57:08 CPU: 0.20% RSS: 23.89GB VMS: 62.60GB[0m | |
| MEMORY: sub proc MPD worker(1781782) increased RSS: rss=573.9MB pss=555.0MB uss=551.4MB shared=22.4MB | |
| MEMORY: sub proc MPD worker(1781783) increased RSS: rss=574.0MB pss=555.0MB uss=551.5MB shared=22.5MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=336.9MB pss=263.8MB uss=211.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=331.6MB pss=258.2MB uss=205.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.8GB uss=21.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=337.0MB pss=263.9MB uss=211.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=331.7MB pss=258.3MB uss=205.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.8GB uss=21.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=337.7MB pss=264.6MB uss=212.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=332.3MB pss=259.0MB uss=206.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.8GB uss=21.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=332.4MB pss=259.0MB uss=206.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.8GB uss=21.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=337.8MB pss=264.7MB uss=212.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=21.8GB uss=21.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=332.5MB pss=259.1MB uss=206.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.8GB uss=21.7GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=19.8GB pss=19.7GB uss=19.7GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=337.9MB pss=264.9MB uss=212.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=332.6MB pss=259.2MB uss=206.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.9GB uss=21.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=338.0MB pss=264.9MB uss=212.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=21.9GB uss=21.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=332.7MB pss=259.3MB uss=206.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.9GB uss=21.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=338.1MB pss=265.0MB uss=212.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=21.9GB uss=21.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=338.2MB pss=265.1MB uss=212.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=332.8MB pss=259.4MB uss=206.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=21.9GB uss=21.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=338.3MB pss=265.2MB uss=212.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=21.9GB uss=21.7GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=19.9GB pss=19.8GB uss=19.8GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=22.0GB uss=21.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=338.4MB pss=265.4MB uss=212.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=333.0MB pss=259.6MB uss=206.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.0GB uss=21.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=338.5MB pss=265.5MB uss=213.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=333.5MB pss=260.2MB uss=207.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.0GB uss=21.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=338.6MB pss=265.5MB uss=213.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.0GB uss=21.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=339.2MB pss=266.1MB uss=213.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.0GB uss=21.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=339.3MB pss=266.2MB uss=213.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=333.8MB pss=260.5MB uss=207.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.0GB uss=21.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=333.9MB pss=260.5MB uss=207.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.0GB uss=21.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=339.4MB pss=266.3MB uss=213.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.0GB uss=21.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=334.0MB pss=260.6MB uss=207.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.0GB uss=21.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=339.5MB pss=266.4MB uss=213.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.0GB uss=21.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=20.0GB pss=19.9GB uss=19.9GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=334.1MB pss=260.7MB uss=207.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.1GB uss=21.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=339.6MB pss=266.5MB uss=214.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.1GB uss=21.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=339.7MB pss=266.6MB uss=214.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=334.2MB pss=260.8MB uss=208.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.1GB uss=21.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=339.8MB pss=266.7MB uss=214.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.1GB uss=21.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=334.3MB pss=260.9MB uss=208.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.1GB uss=21.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=339.9MB pss=266.8MB uss=214.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.1GB uss=21.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=340.1MB pss=267.0MB uss=214.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=334.4MB pss=261.1MB uss=208.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.1GB uss=21.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=334.5MB pss=261.1MB uss=208.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.1GB uss=21.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=20.1GB pss=20.0GB uss=20.0GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=340.7MB pss=267.6MB uss=215.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=334.6MB pss=261.2MB uss=208.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.2GB uss=22.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=340.8MB pss=267.7MB uss=215.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=334.7MB pss=261.3MB uss=208.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.2GB uss=22.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=335.3MB pss=261.9MB uss=209.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.2GB uss=22.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=335.7MB pss=262.3MB uss=209.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.2GB uss=22.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=340.9MB pss=267.8MB uss=215.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=335.8MB pss=262.4MB uss=209.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.2GB uss=22.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=341.0MB pss=267.9MB uss=215.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=335.9MB pss=262.5MB uss=209.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.2GB uss=22.0GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=20.2GB pss=20.1GB uss=20.1GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=341.1MB pss=268.0MB uss=215.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.3GB uss=22.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=341.2MB pss=268.1MB uss=215.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.3GB uss=22.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=341.3MB pss=268.2MB uss=215.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=336.0MB pss=262.7MB uss=209.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.3GB uss=22.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=341.4MB pss=268.3MB uss=215.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=336.6MB pss=263.2MB uss=210.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.3GB uss=22.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=341.5MB pss=268.4MB uss=215.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.3GB uss=22.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=341.6MB pss=268.5MB uss=216.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.3GB uss=22.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=342.2MB pss=269.1MB uss=216.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=336.7MB pss=263.3MB uss=210.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.3GB uss=22.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=342.3MB pss=269.2MB uss=216.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.3GB uss=22.1GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=20.3GB pss=20.2GB uss=20.2GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=342.4MB pss=269.3MB uss=216.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=336.8MB pss=263.4MB uss=210.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.4GB uss=22.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=336.9MB pss=263.5MB uss=210.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.4GB uss=22.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=342.5MB pss=269.4MB uss=216.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=337.0MB pss=263.6MB uss=210.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.4GB uss=22.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=337.1MB pss=263.7MB uss=210.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.4GB uss=22.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=342.7MB pss=269.6MB uss=217.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.4GB uss=22.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=342.8MB pss=269.7MB uss=217.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.4GB uss=22.2GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=20.4GB pss=20.3GB uss=20.3GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=337.2MB pss=263.8MB uss=211.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.5GB uss=22.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=343.0MB pss=269.9MB uss=217.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.5GB uss=22.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=343.1MB pss=270.0MB uss=217.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=337.3MB pss=263.9MB uss=211.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.5GB uss=22.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=343.8MB pss=270.7MB uss=218.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=337.4MB pss=264.0MB uss=211.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.5GB uss=22.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=343.9MB pss=270.8MB uss=218.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=337.5MB pss=264.1MB uss=211.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.5GB uss=22.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=344.0MB pss=270.9MB uss=218.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=337.6MB pss=264.3MB uss=211.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.5GB uss=22.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=337.7MB pss=264.3MB uss=211.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.5GB uss=22.3GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=20.5GB pss=20.4GB uss=20.4GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=22.6GB uss=22.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=337.8MB pss=264.4MB uss=211.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.6GB uss=22.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=344.1MB pss=271.0MB uss=218.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=338.4MB pss=265.0MB uss=212.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.6GB uss=22.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=338.8MB pss=265.5MB uss=212.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.6GB uss=22.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=338.9MB pss=265.5MB uss=212.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.6GB uss=22.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=344.2MB pss=271.1MB uss=218.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.6GB uss=22.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=344.3MB pss=271.2MB uss=218.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.6GB uss=22.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=344.4MB pss=271.3MB uss=218.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=339.0MB pss=265.6MB uss=212.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.6GB uss=22.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=344.5MB pss=271.4MB uss=218.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.6GB uss=22.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=344.6MB pss=271.5MB uss=219.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.6GB uss=22.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=344.7MB pss=271.6MB uss=219.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=339.1MB pss=265.7MB uss=212.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.6GB uss=22.4GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=20.6GB pss=20.5GB uss=20.5GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=22.7GB uss=22.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=345.3MB pss=272.2MB uss=219.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=339.6MB pss=266.2MB uss=213.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.7GB uss=22.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=345.4MB pss=272.3MB uss=219.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.7GB uss=22.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=339.7MB pss=266.3MB uss=213.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.7GB uss=22.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=345.5MB pss=272.4MB uss=219.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.7GB uss=22.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=345.6MB pss=272.5MB uss=220.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.7GB uss=22.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=345.7MB pss=272.6MB uss=220.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.7GB uss=22.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=339.8MB pss=266.4MB uss=213.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.7GB uss=22.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=345.8MB pss=272.7MB uss=220.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.7GB uss=22.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=345.9MB pss=272.8MB uss=220.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.7GB uss=22.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=339.9MB pss=266.5MB uss=213.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.7GB uss=22.6GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=20.7GB pss=20.6GB uss=20.6GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=22.8GB uss=22.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=346.0MB pss=272.9MB uss=220.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=340.0MB pss=266.6MB uss=213.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.8GB uss=22.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=340.5MB pss=267.2MB uss=214.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.8GB uss=22.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=346.1MB pss=273.0MB uss=220.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=340.6MB pss=267.2MB uss=214.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.8GB uss=22.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=346.2MB pss=273.1MB uss=220.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.8GB uss=22.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=347.1MB pss=274.1MB uss=221.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=340.7MB pss=267.3MB uss=214.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.8GB uss=22.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=347.2MB pss=274.1MB uss=221.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.8GB uss=22.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=340.8MB pss=267.4MB uss=214.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.8GB uss=22.7GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=20.8GB pss=20.7GB uss=20.7GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=22.9GB uss=22.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=347.3MB pss=274.2MB uss=221.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.9GB uss=22.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=340.9MB pss=267.5MB uss=214.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.9GB uss=22.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=347.4MB pss=274.3MB uss=221.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.9GB uss=22.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=341.0MB pss=267.6MB uss=214.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.9GB uss=22.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=347.5MB pss=274.4MB uss=221.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.9GB uss=22.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=341.1MB pss=267.7MB uss=214.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=22.9GB uss=22.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=347.6MB pss=274.5MB uss=222.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=22.9GB uss=22.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=20.9GB pss=20.8GB uss=20.8GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=341.2MB pss=267.8MB uss=215.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.0GB uss=22.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=347.7MB pss=274.6MB uss=222.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.0GB uss=22.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=348.2MB pss=275.2MB uss=222.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=341.4MB pss=268.0MB uss=215.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.0GB uss=22.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=348.7MB pss=275.6MB uss=223.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=341.5MB pss=268.1MB uss=215.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.0GB uss=22.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=348.8MB pss=275.7MB uss=223.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=342.2MB pss=268.8MB uss=216.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.0GB uss=22.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=348.9MB pss=275.8MB uss=223.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=342.3MB pss=268.9MB uss=216.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.0GB uss=22.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=342.4MB pss=269.1MB uss=216.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.0GB uss=22.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=21.0GB pss=20.9GB uss=20.9GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=342.5MB pss=269.1MB uss=216.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.1GB uss=23.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=349.0MB pss=275.9MB uss=223.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.1GB uss=23.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=342.6MB pss=269.2MB uss=216.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.1GB uss=23.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=349.1MB pss=276.0MB uss=223.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=342.7MB pss=269.3MB uss=216.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.1GB uss=23.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=349.2MB pss=276.1MB uss=223.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.1GB uss=23.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=342.8MB pss=269.4MB uss=216.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.1GB uss=23.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=349.3MB pss=276.2MB uss=223.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.1GB uss=23.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=349.4MB pss=276.3MB uss=223.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=342.9MB pss=269.5MB uss=216.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.1GB uss=23.0GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=21.1GB pss=21.0GB uss=21.0GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=349.5MB pss=276.4MB uss=223.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.2GB uss=23.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=343.0MB pss=269.6MB uss=216.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.2GB uss=23.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=349.7MB pss=276.6MB uss=224.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=343.1MB pss=269.8MB uss=217.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.2GB uss=23.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=343.7MB pss=270.3MB uss=217.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.2GB uss=23.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=349.8MB pss=276.7MB uss=224.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.2GB uss=23.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=344.1MB pss=270.8MB uss=218.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.2GB uss=23.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=350.0MB pss=276.9MB uss=224.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=344.2MB pss=270.8MB uss=218.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.2GB uss=23.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=350.5MB pss=277.4MB uss=224.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.2GB uss=23.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=350.6MB pss=277.5MB uss=225.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.2GB uss=23.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=350.7MB pss=277.6MB uss=225.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=344.3MB pss=270.9MB uss=218.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.2GB uss=23.1GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=21.2GB pss=21.1GB uss=21.1GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=23.3GB uss=23.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=350.8MB pss=277.7MB uss=225.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=344.4MB pss=271.0MB uss=218.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.3GB uss=23.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=344.5MB pss=271.1MB uss=218.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.3GB uss=23.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=344.6MB pss=271.2MB uss=218.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.3GB uss=23.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=350.9MB pss=277.8MB uss=225.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.3GB uss=23.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=351.0MB pss=277.9MB uss=225.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=345.1MB pss=271.8MB uss=219.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.3GB uss=23.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=351.1MB pss=278.0MB uss=225.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.3GB uss=23.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=345.2MB pss=271.8MB uss=219.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.3GB uss=23.2GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=21.3GB pss=21.2GB uss=21.2GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=351.3MB pss=278.2MB uss=225.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.4GB uss=23.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=351.4MB pss=278.3MB uss=225.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.4GB uss=23.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=345.3MB pss=271.9MB uss=219.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.4GB uss=23.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=352.1MB pss=279.0MB uss=226.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.4GB uss=23.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=345.4MB pss=272.0MB uss=219.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.4GB uss=23.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=352.2MB pss=279.1MB uss=226.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.4GB uss=23.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=345.5MB pss=272.1MB uss=219.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.4GB uss=23.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=352.3MB pss=279.3MB uss=226.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=345.6MB pss=272.2MB uss=219.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.4GB uss=23.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=352.4MB pss=279.3MB uss=226.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.4GB uss=23.3GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=21.4GB pss=21.3GB uss=21.3GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=345.7MB pss=272.3MB uss=219.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.5GB uss=23.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=352.5MB pss=279.4MB uss=226.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.5GB uss=23.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=345.8MB pss=272.4MB uss=219.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.5GB uss=23.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=352.6MB pss=279.6MB uss=227.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.5GB uss=23.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=352.7MB pss=279.6MB uss=227.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.5GB uss=23.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=352.9MB pss=279.8MB uss=227.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=346.0MB pss=272.7MB uss=219.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.5GB uss=23.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=346.1MB pss=272.7MB uss=219.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.5GB uss=23.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=353.0MB pss=279.9MB uss=227.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.5GB uss=23.4GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=21.5GB pss=21.4GB uss=21.4GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=23.6GB uss=23.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=354.1MB pss=281.0MB uss=228.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=347.2MB pss=273.8MB uss=221.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.6GB uss=23.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=354.2MB pss=281.1MB uss=228.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=347.3MB pss=273.9MB uss=221.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.6GB uss=23.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=354.3MB pss=281.2MB uss=228.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=347.4MB pss=274.0MB uss=221.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.6GB uss=23.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=354.4MB pss=281.3MB uss=228.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.6GB uss=23.5GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=21.6GB pss=21.5GB uss=21.5GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=23.7GB uss=23.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=354.5MB pss=281.4MB uss=228.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=347.5MB pss=274.2MB uss=221.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.7GB uss=23.6GB | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=7.5GB, std_dev=0.0B, min=7.5GB, max=7.5GB, num_seqs=4143, avg_data_len=1) | |
| train epoch 67, finished after 4143 steps, 0:19:52 elapsed (99.4% computing time) | |
| Save model under /u/maximilian.kannen/setups/20230406_feat/work/i6_core/returnn/training/ReturnnTrainingJob.wQIA3Rc1nLak/output/models/epoch.067 | |
| epoch 67 score: 1.2387008987911692 error: None elapsed: 0:19:52 | |
| MEMORY: main proc python3(1781720) increased RSS: rss=21.7GB pss=21.7GB uss=21.7GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=354.6MB pss=281.5MB uss=229.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=347.6MB pss=274.2MB uss=221.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.9GB uss=23.8GB | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=7.5GB, std_dev=0.0B, min=7.5GB, max=7.5GB, num_seqs=31, avg_data_len=1) | |
| epoch 67 'dev' eval, finished after 31 steps, 0:00:03 elapsed (62.3% computing time) | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=7.5GB, std_dev=0.0B, min=7.5GB, max=7.5GB, num_seqs=29, avg_data_len=1) | |
| epoch 67 'devtrain' eval, finished after 29 steps, 0:00:03 elapsed (61.9% computing time) | |
| dev: score 1.6138407588263644 error None devtrain: score 1.5968489763866855 error None | |
| We have stored models for epochs [4, 5, 6, ..., 65, 66, 67] and keep epochs [4, 5, 6, 7, 8, 9, 10, 11, 63, 64, 65, 66, 67]. | |
| We will delete the models of epochs [62]. | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=355.1MB pss=282.0MB uss=229.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=348.2MB pss=274.9MB uss=222.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.9GB uss=23.8GB | |
| Deleted 335.5MB. | |
| start epoch 68 with learning rate 0.00015720694444444443 ... | |
| MEMORY: sub proc MPD seq order(1781781) increased RSS: rss=573.3MB pss=555.2MB uss=551.8MB shared=21.5MB | |
| MEMORY: sub proc MPD worker(1781782) increased RSS: rss=574.0MB pss=555.1MB uss=551.5MB shared=22.4MB | |
| MEMORY: sub proc MPD worker(1781783) increased RSS: rss=574.1MB pss=555.1MB uss=551.6MB shared=22.5MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=348.3MB pss=274.9MB uss=222.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.9GB uss=23.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=355.2MB pss=282.1MB uss=229.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.9GB uss=23.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=348.4MB pss=275.0MB uss=222.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.9GB uss=23.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=355.3MB pss=282.2MB uss=229.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.9GB uss=23.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=355.4MB pss=282.4MB uss=229.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=348.5MB pss=275.1MB uss=222.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.9GB uss=23.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=355.5MB pss=282.4MB uss=229.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.9GB uss=23.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=348.6MB pss=275.2MB uss=222.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.9GB uss=23.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=355.6MB pss=282.5MB uss=230.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=348.7MB pss=275.3MB uss=222.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.9GB uss=23.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=355.7MB pss=282.6MB uss=230.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.9GB uss=23.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=355.8MB pss=282.7MB uss=230.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=348.8MB pss=275.4MB uss=222.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.9GB uss=23.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=356.0MB pss=282.9MB uss=230.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=348.9MB pss=275.5MB uss=222.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.9GB uss=23.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=356.5MB pss=283.5MB uss=231.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.9GB uss=23.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=356.6MB pss=283.5MB uss=231.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=349.0MB pss=275.6MB uss=222.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.9GB uss=23.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=21.8GB pss=21.7GB uss=21.7GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=23.9GB uss=23.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=356.7MB pss=283.6MB uss=231.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=349.2MB pss=275.8MB uss=223.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.9GB uss=23.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=349.7MB pss=276.3MB uss=223.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.9GB uss=23.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=356.8MB pss=283.7MB uss=231.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=349.8MB pss=276.4MB uss=223.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.9GB uss=23.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=357.0MB pss=283.9MB uss=231.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=349.9MB pss=276.5MB uss=223.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=23.9GB uss=23.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=357.1MB pss=284.0MB uss=231.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=23.9GB uss=23.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=21.9GB pss=21.8GB uss=21.8GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=24.0GB uss=23.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=357.3MB pss=284.2MB uss=231.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=350.0MB pss=276.6MB uss=223.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.0GB uss=23.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=350.1MB pss=276.7MB uss=223.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.0GB uss=23.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=357.4MB pss=284.3MB uss=231.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.0GB uss=23.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=358.1MB pss=285.0MB uss=232.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=350.3MB pss=276.9MB uss=224.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.0GB uss=23.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=350.4MB pss=277.0MB uss=224.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.0GB uss=23.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=358.2MB pss=285.1MB uss=232.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.0GB uss=23.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=358.4MB pss=285.3MB uss=232.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=350.6MB pss=277.2MB uss=224.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.0GB uss=23.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=358.5MB pss=285.4MB uss=232.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.0GB uss=23.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=351.1MB pss=277.8MB uss=225.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.0GB uss=23.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=22.0GB pss=21.9GB uss=21.9GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=351.2MB pss=277.8MB uss=225.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.1GB uss=24.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=358.6MB pss=285.5MB uss=233.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=351.7MB pss=278.3MB uss=225.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.1GB uss=24.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=358.7MB pss=285.6MB uss=233.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=351.8MB pss=278.4MB uss=225.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.1GB uss=24.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=358.8MB pss=285.7MB uss=233.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.1GB uss=24.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=358.9MB pss=285.8MB uss=233.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=351.9MB pss=278.5MB uss=225.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.1GB uss=24.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=352.0MB pss=278.6MB uss=225.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.1GB uss=24.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=359.0MB pss=285.9MB uss=233.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.1GB uss=24.0GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=22.1GB pss=22.0GB uss=22.0GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=359.6MB pss=286.6MB uss=234.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=352.1MB pss=278.7MB uss=226.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.2GB uss=24.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=359.7MB pss=286.6MB uss=234.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.2GB uss=24.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=359.8MB pss=286.7MB uss=234.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=352.2MB pss=278.8MB uss=226.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.2GB uss=24.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=359.9MB pss=286.8MB uss=234.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.2GB uss=24.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=360.0MB pss=287.0MB uss=234.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.2GB uss=24.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=360.1MB pss=287.0MB uss=234.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.2GB uss=24.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=352.3MB pss=278.9MB uss=226.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.2GB uss=24.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=360.2MB pss=287.1MB uss=234.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=352.8MB pss=279.4MB uss=226.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.2GB uss=24.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=360.3MB pss=287.2MB uss=234.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.2GB uss=24.1GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=22.2GB pss=22.1GB uss=22.1GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=24.3GB uss=24.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=360.4MB pss=287.3MB uss=234.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.3GB uss=24.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=360.5MB pss=287.4MB uss=234.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=352.9MB pss=279.5MB uss=226.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.3GB uss=24.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=360.6MB pss=287.5MB uss=235.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=353.4MB pss=280.1MB uss=227.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.3GB uss=24.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=361.1MB pss=288.0MB uss=235.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.3GB uss=24.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=361.2MB pss=288.1MB uss=235.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=353.5MB pss=280.1MB uss=227.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.3GB uss=24.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=353.6MB pss=280.2MB uss=227.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.3GB uss=24.2GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=22.3GB pss=22.2GB uss=22.2GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=361.3MB pss=288.2MB uss=235.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.4GB uss=24.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=353.7MB pss=280.3MB uss=227.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.4GB uss=24.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=361.4MB pss=288.3MB uss=235.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.4GB uss=24.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=361.5MB pss=288.4MB uss=235.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=353.8MB pss=280.4MB uss=227.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.4GB uss=24.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=361.6MB pss=288.5MB uss=236.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.4GB uss=24.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=353.9MB pss=280.5MB uss=227.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.4GB uss=24.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=361.7MB pss=288.6MB uss=236.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=354.0MB pss=280.6MB uss=227.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.4GB uss=24.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=361.8MB pss=288.7MB uss=236.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=354.1MB pss=280.7MB uss=227.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.4GB uss=24.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=361.9MB pss=288.8MB uss=236.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.4GB uss=24.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=362.5MB pss=289.5MB uss=237.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=354.3MB pss=280.9MB uss=228.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.4GB uss=24.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=362.6MB pss=289.5MB uss=237.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=354.4MB pss=281.0MB uss=228.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.4GB uss=24.3GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=22.4GB pss=22.3GB uss=22.3GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=362.7MB pss=289.6MB uss=237.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=354.9MB pss=281.5MB uss=228.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.5GB uss=24.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=355.3MB pss=281.9MB uss=229.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.5GB uss=24.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=362.8MB pss=289.7MB uss=237.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=355.4MB pss=282.0MB uss=229.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.5GB uss=24.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=362.9MB pss=289.8MB uss=237.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=355.5MB pss=282.1MB uss=229.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.5GB uss=24.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=363.0MB pss=289.9MB uss=237.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=355.6MB pss=282.2MB uss=229.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.5GB uss=24.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=363.1MB pss=290.0MB uss=237.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=355.7MB pss=282.3MB uss=229.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.5GB uss=24.4GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=22.5GB pss=22.4GB uss=22.4GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=24.6GB uss=24.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=363.2MB pss=290.1MB uss=237.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=355.8MB pss=282.4MB uss=229.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.6GB uss=24.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=363.3MB pss=290.3MB uss=237.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=356.3MB pss=283.0MB uss=230.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.6GB uss=24.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=363.4MB pss=290.4MB uss=237.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=356.4MB pss=283.0MB uss=230.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.6GB uss=24.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=363.5MB pss=290.4MB uss=237.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.6GB uss=24.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=364.1MB pss=291.0MB uss=238.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.6GB uss=24.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=364.6MB pss=291.5MB uss=239.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=356.5MB pss=283.1MB uss=230.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.6GB uss=24.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=364.7MB pss=291.6MB uss=239.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=356.6MB pss=283.2MB uss=230.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.6GB uss=24.5GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=22.6GB pss=22.5GB uss=22.5GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=24.7GB uss=24.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=356.7MB pss=283.3MB uss=230.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.7GB uss=24.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=364.8MB pss=291.8MB uss=239.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.7GB uss=24.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=365.3MB pss=292.2MB uss=239.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=356.8MB pss=283.4MB uss=230.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.7GB uss=24.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=365.4MB pss=292.3MB uss=239.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.7GB uss=24.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=356.9MB pss=283.5MB uss=230.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.7GB uss=24.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=365.5MB pss=292.4MB uss=239.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.7GB uss=24.6GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=22.7GB pss=22.6GB uss=22.6GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=357.0MB pss=283.6MB uss=230.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.8GB uss=24.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=365.6MB pss=292.5MB uss=240.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.8GB uss=24.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=365.7MB pss=292.6MB uss=240.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=357.2MB pss=283.8MB uss=231.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.8GB uss=24.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=357.3MB pss=283.9MB uss=231.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.8GB uss=24.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=365.8MB pss=292.7MB uss=240.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=358.0MB pss=284.6MB uss=231.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.8GB uss=24.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=358.1MB pss=284.7MB uss=231.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.8GB uss=24.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=365.9MB pss=292.9MB uss=240.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.8GB uss=24.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=366.0MB pss=293.0MB uss=240.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.8GB uss=24.7GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=22.8GB pss=22.7GB uss=22.7GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=366.1MB pss=293.0MB uss=240.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=358.2MB pss=284.8MB uss=232.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.9GB uss=24.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=366.3MB pss=293.2MB uss=240.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.9GB uss=24.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=366.4MB pss=293.3MB uss=240.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.9GB uss=24.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=358.3MB pss=284.9MB uss=232.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.9GB uss=24.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=366.5MB pss=293.5MB uss=241.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.9GB uss=24.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=367.1MB pss=294.0MB uss=241.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=358.4MB pss=285.0MB uss=232.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.9GB uss=24.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=367.5MB pss=294.5MB uss=242.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=358.5MB pss=285.1MB uss=232.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.9GB uss=24.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=367.6MB pss=294.5MB uss=242.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=24.9GB uss=24.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=358.6MB pss=285.2MB uss=232.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.9GB uss=24.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=358.7MB pss=285.3MB uss=232.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=24.9GB uss=24.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=22.9GB pss=22.8GB uss=22.8GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=367.7MB pss=294.6MB uss=242.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=358.8MB pss=285.4MB uss=232.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.0GB uss=24.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=358.9MB pss=285.5MB uss=232.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.0GB uss=24.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=367.8MB pss=294.7MB uss=242.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=359.4MB pss=286.1MB uss=233.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.0GB uss=24.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=359.5MB pss=286.1MB uss=233.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.0GB uss=24.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=368.4MB pss=295.3MB uss=242.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=359.6MB pss=286.2MB uss=233.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.0GB uss=24.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=359.7MB pss=286.3MB uss=233.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.0GB uss=24.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=368.5MB pss=295.4MB uss=242.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.0GB uss=24.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=359.8MB pss=286.4MB uss=233.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.0GB uss=24.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=23.0GB pss=22.9GB uss=22.9GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=368.6MB pss=295.6MB uss=243.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.1GB uss=25.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=368.7MB pss=295.6MB uss=243.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=359.9MB pss=286.5MB uss=233.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.1GB uss=25.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=368.8MB pss=295.7MB uss=243.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.1GB uss=25.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=368.9MB pss=295.8MB uss=243.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=360.0MB pss=286.7MB uss=233.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.1GB uss=25.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=360.1MB pss=286.8MB uss=234.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.1GB uss=25.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=360.2MB pss=286.8MB uss=234.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.1GB uss=25.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=369.0MB pss=295.9MB uss=243.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=360.3MB pss=286.9MB uss=234.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.1GB uss=25.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=360.4MB pss=287.0MB uss=234.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.1GB uss=25.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=360.9MB pss=287.5MB uss=234.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.1GB uss=25.0GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=23.1GB pss=23.0GB uss=23.0GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=369.1MB pss=296.0MB uss=243.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=361.0MB pss=287.7MB uss=234.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.2GB uss=25.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=369.2MB pss=296.1MB uss=243.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=361.1MB pss=287.7MB uss=234.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.2GB uss=25.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=369.3MB pss=296.2MB uss=243.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.2GB uss=25.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=369.4MB pss=296.3MB uss=243.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.2GB uss=25.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=369.5MB pss=296.4MB uss=243.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=361.2MB pss=287.8MB uss=235.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.2GB uss=25.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=369.6MB pss=296.5MB uss=244.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.2GB uss=25.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=370.1MB pss=297.0MB uss=244.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.2GB uss=25.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=370.2MB pss=297.1MB uss=244.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.2GB uss=25.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=361.3MB pss=288.0MB uss=235.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.2GB uss=25.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=370.3MB pss=297.2MB uss=244.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=361.4MB pss=288.1MB uss=235.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.2GB uss=25.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=361.6MB pss=288.2MB uss=235.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.2GB uss=25.1GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=23.2GB pss=23.1GB uss=23.1GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=25.3GB uss=25.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=370.4MB pss=297.3MB uss=244.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=361.7MB pss=288.4MB uss=235.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.3GB uss=25.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=370.5MB pss=297.4MB uss=244.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=361.9MB pss=288.5MB uss=235.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.3GB uss=25.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=362.4MB pss=289.0MB uss=236.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.3GB uss=25.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=370.6MB pss=297.5MB uss=245.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.3GB uss=25.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=370.7MB pss=297.6MB uss=245.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=362.5MB pss=289.1MB uss=236.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.3GB uss=25.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=370.8MB pss=297.7MB uss=245.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.3GB uss=25.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=370.9MB pss=297.8MB uss=245.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.3GB uss=25.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=371.0MB pss=298.0MB uss=245.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=362.6MB pss=289.2MB uss=236.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.3GB uss=25.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=371.6MB pss=298.5MB uss=246.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.3GB uss=25.2GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=23.3GB pss=23.2GB uss=23.2GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=371.7MB pss=298.6MB uss=246.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.4GB uss=25.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=371.8MB pss=298.7MB uss=246.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=362.7MB pss=289.3MB uss=236.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.4GB uss=25.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=362.8MB pss=289.4MB uss=236.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.4GB uss=25.3GB | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=7.5GB, std_dev=0.0B, min=7.5GB, max=7.5GB, num_seqs=4178, avg_data_len=1) | |
| train epoch 68, finished after 4178 steps, 0:19:40 elapsed (99.4% computing time) | |
| Save model under /u/maximilian.kannen/setups/20230406_feat/work/i6_core/returnn/training/ReturnnTrainingJob.wQIA3Rc1nLak/output/models/epoch.068 | |
| MEMORY: main proc python3(1781720) increased RSS: rss=23.5GB pss=23.4GB uss=23.4GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=371.9MB pss=298.8MB uss=246.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=362.9MB pss=289.6MB uss=236.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.6GB uss=25.5GB | |
| epoch 68 score: 1.2434659071171972 error: None elapsed: 0:19:40 | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=363.0MB pss=289.6MB uss=236.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.6GB uss=25.5GB | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=7.5GB, std_dev=0.0B, min=7.5GB, max=7.5GB, num_seqs=31, avg_data_len=1) | |
| epoch 68 'dev' eval, finished after 31 steps, 0:00:03 elapsed (63.6% computing time) | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=7.5GB, std_dev=0.0B, min=7.5GB, max=7.5GB, num_seqs=29, avg_data_len=1) | |
| epoch 68 'devtrain' eval, finished after 29 steps, 0:00:03 elapsed (61.8% computing time) | |
| dev: score 1.5910986359433221 error None devtrain: score 1.5743923734123488 error None | |
| We have stored models for epochs [4, 5, 6, ..., 66, 67, 68] and keep epochs [4, 5, 6, 7, 8, 9, 10, 11, 64, 65, 66, 67, 68]. | |
| We will delete the models of epochs [63]. | |
| Deleted 335.5MB. | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=363.1MB pss=289.8MB uss=237.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.6GB uss=25.5GB | |
| start epoch 69 with learning rate 0.00015935555555555555 ... | |
| [2023-10-31 02:28:57,516] INFO: [32mRun time: 2:24:17 CPU: 0.20% RSS: 26.28GB VMS: 65.79GB[0m | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=372.0MB pss=298.9MB uss=246.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=363.2MB pss=289.8MB uss=237.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.6GB uss=25.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=363.3MB pss=289.9MB uss=237.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.6GB uss=25.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=372.1MB pss=299.0MB uss=246.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.6GB uss=25.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=372.3MB pss=299.2MB uss=246.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=364.0MB pss=290.6MB uss=237.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.6GB uss=25.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=372.4MB pss=299.3MB uss=246.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.6GB uss=25.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=373.1MB pss=300.0MB uss=247.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=364.1MB pss=290.7MB uss=237.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.6GB uss=25.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=373.2MB pss=300.1MB uss=247.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.6GB uss=25.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=373.3MB pss=300.2MB uss=247.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=364.2MB pss=290.8MB uss=238.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.6GB uss=25.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=373.4MB pss=300.3MB uss=247.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.6GB uss=25.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=364.4MB pss=291.0MB uss=238.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.6GB uss=25.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=373.5MB pss=300.4MB uss=247.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=364.5MB pss=291.1MB uss=238.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.6GB uss=25.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=373.6MB pss=300.5MB uss=248.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=364.6MB pss=291.3MB uss=238.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.6GB uss=25.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=364.7MB pss=291.3MB uss=238.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.6GB uss=25.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=373.7MB pss=300.6MB uss=248.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.6GB uss=25.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=373.9MB pss=300.8MB uss=248.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=365.4MB pss=292.0MB uss=239.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.6GB uss=25.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=374.0MB pss=300.9MB uss=248.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.6GB uss=25.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=365.5MB pss=292.1MB uss=239.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.6GB uss=25.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=375.1MB pss=302.0MB uss=249.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=366.1MB pss=292.7MB uss=239.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.6GB uss=25.5GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=23.6GB pss=23.5GB uss=23.5GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=375.2MB pss=302.1MB uss=249.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.7GB uss=25.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=375.3MB pss=302.2MB uss=249.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.7GB uss=25.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=375.4MB pss=302.3MB uss=249.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.7GB uss=25.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=376.0MB pss=302.9MB uss=250.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.7GB uss=25.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=366.2MB pss=292.8MB uss=240.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.7GB uss=25.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=376.1MB pss=303.0MB uss=250.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=366.4MB pss=293.0MB uss=240.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.7GB uss=25.6GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=23.7GB pss=23.6GB uss=23.6GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=366.9MB pss=293.5MB uss=240.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.8GB uss=25.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=376.2MB pss=303.1MB uss=250.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=367.0MB pss=293.6MB uss=240.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.8GB uss=25.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=376.3MB pss=303.2MB uss=250.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=367.5MB pss=294.1MB uss=241.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.8GB uss=25.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=376.4MB pss=303.3MB uss=250.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.8GB uss=25.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=376.5MB pss=303.4MB uss=250.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=367.6MB pss=294.2MB uss=241.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.8GB uss=25.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=376.6MB pss=303.5MB uss=251.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.8GB uss=25.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=368.2MB pss=294.8MB uss=242.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.8GB uss=25.7GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=23.8GB pss=23.7GB uss=23.7GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=25.9GB uss=25.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=376.7MB pss=303.6MB uss=251.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.9GB uss=25.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=376.8MB pss=303.7MB uss=251.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.9GB uss=25.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=368.3MB pss=294.9MB uss=242.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.9GB uss=25.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=376.9MB pss=303.8MB uss=251.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.9GB uss=25.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=377.1MB pss=304.0MB uss=251.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.9GB uss=25.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=378.0MB pss=305.0MB uss=252.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=368.4MB pss=295.0MB uss=242.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.9GB uss=25.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=378.1MB pss=305.0MB uss=252.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=25.9GB uss=25.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=368.9MB pss=295.5MB uss=242.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=25.9GB uss=25.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=23.9GB pss=23.8GB uss=23.8GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=378.2MB pss=305.1MB uss=252.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.0GB uss=25.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=369.0MB pss=295.6MB uss=242.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.0GB uss=25.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=378.3MB pss=305.2MB uss=252.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.0GB uss=25.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=378.9MB pss=305.8MB uss=253.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=369.1MB pss=295.7MB uss=242.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.0GB uss=25.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=369.2MB pss=295.8MB uss=243.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.0GB uss=25.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=369.3MB pss=295.9MB uss=243.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.0GB uss=25.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=379.0MB pss=305.9MB uss=253.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=369.8MB pss=296.4MB uss=243.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.0GB uss=25.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=369.9MB pss=296.5MB uss=243.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.0GB uss=25.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=379.6MB pss=306.5MB uss=254.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.0GB uss=25.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=24.0GB pss=23.9GB uss=23.9GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=26.1GB uss=26.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=379.7MB pss=306.6MB uss=254.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=370.0MB pss=296.6MB uss=243.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.1GB uss=26.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=370.1MB pss=296.7MB uss=243.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.1GB uss=26.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=379.8MB pss=306.7MB uss=254.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=370.2MB pss=296.9MB uss=244.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.1GB uss=26.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=379.9MB pss=306.8MB uss=254.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=370.3MB pss=296.9MB uss=244.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.1GB uss=26.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=380.3MB pss=307.2MB uss=254.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.1GB uss=26.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=380.4MB pss=307.3MB uss=254.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.1GB uss=26.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=370.4MB pss=297.0MB uss=244.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.1GB uss=26.0GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=24.1GB pss=24.0GB uss=24.0GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=26.2GB uss=26.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=380.5MB pss=307.4MB uss=254.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.2GB uss=26.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=370.5MB pss=297.1MB uss=244.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.2GB uss=26.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=380.6MB pss=307.5MB uss=255.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.2GB uss=26.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=381.1MB pss=308.1MB uss=255.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=370.7MB pss=297.3MB uss=244.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.2GB uss=26.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=381.2MB pss=308.1MB uss=255.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.2GB uss=26.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=370.8MB pss=297.4MB uss=244.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.2GB uss=26.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=371.5MB pss=298.1MB uss=245.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.2GB uss=26.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=381.3MB pss=308.2MB uss=255.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=371.9MB pss=298.6MB uss=245.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.2GB uss=26.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=372.0MB pss=298.6MB uss=245.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.2GB uss=26.1GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=24.2GB pss=24.1GB uss=24.1GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=381.4MB pss=308.3MB uss=255.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.3GB uss=26.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=381.5MB pss=308.4MB uss=255.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.3GB uss=26.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=372.1MB pss=298.7MB uss=245.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.3GB uss=26.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=382.1MB pss=309.1MB uss=256.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.3GB uss=26.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=382.2MB pss=309.1MB uss=256.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.3GB uss=26.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=372.2MB pss=298.8MB uss=246.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.3GB uss=26.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=372.3MB pss=299.0MB uss=246.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.3GB uss=26.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=372.4MB pss=299.0MB uss=246.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.3GB uss=26.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=382.3MB pss=309.2MB uss=256.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.3GB uss=26.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=372.5MB pss=299.1MB uss=246.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.3GB uss=26.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=382.4MB pss=309.3MB uss=256.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=373.0MB pss=299.6MB uss=246.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.3GB uss=26.2GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=24.3GB pss=24.2GB uss=24.2GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=26.4GB uss=26.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=382.5MB pss=309.4MB uss=256.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.4GB uss=26.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=373.1MB pss=299.7MB uss=246.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.4GB uss=26.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=382.6MB pss=309.5MB uss=257.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.4GB uss=26.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=382.7MB pss=309.6MB uss=257.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=373.2MB pss=299.8MB uss=247.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.4GB uss=26.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=373.3MB pss=299.9MB uss=247.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.4GB uss=26.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=382.8MB pss=309.8MB uss=257.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=373.4MB pss=300.1MB uss=247.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.4GB uss=26.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=382.9MB pss=309.8MB uss=257.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=373.5MB pss=300.2MB uss=247.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.4GB uss=26.3GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=24.4GB pss=24.3GB uss=24.3GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=383.0MB pss=309.9MB uss=257.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=373.6MB pss=300.2MB uss=247.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.5GB uss=26.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=383.6MB pss=310.5MB uss=258.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=373.7MB pss=300.3MB uss=247.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.5GB uss=26.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=383.7MB pss=310.6MB uss=258.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.5GB uss=26.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=383.8MB pss=310.7MB uss=258.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.5GB uss=26.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=383.9MB pss=310.8MB uss=258.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=373.8MB pss=300.4MB uss=247.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.5GB uss=26.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=384.0MB pss=310.9MB uss=258.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=373.9MB pss=300.5MB uss=247.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.5GB uss=26.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=374.0MB pss=300.6MB uss=247.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.5GB uss=26.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=384.1MB pss=311.0MB uss=258.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=374.6MB pss=301.2MB uss=248.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.5GB uss=26.4GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=24.5GB pss=24.4GB uss=24.4GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=384.2MB pss=311.1MB uss=258.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.7GB uss=26.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=384.3MB pss=311.2MB uss=258.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=374.7MB pss=301.3MB uss=248.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.7GB uss=26.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=384.4MB pss=311.3MB uss=258.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.7GB uss=26.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=384.5MB pss=311.4MB uss=258.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.7GB uss=26.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=384.6MB pss=311.5MB uss=259.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.7GB uss=26.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=374.8MB pss=301.4MB uss=248.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.7GB uss=26.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=385.6MB pss=312.6MB uss=260.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.7GB uss=26.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=385.7MB pss=312.6MB uss=260.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.7GB uss=26.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=374.9MB pss=301.5MB uss=248.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.7GB uss=26.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=375.0MB pss=301.6MB uss=248.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.7GB uss=26.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=385.8MB pss=312.7MB uss=260.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=375.1MB pss=301.7MB uss=248.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.7GB uss=26.5GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=24.6GB pss=24.5GB uss=24.5GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=26.8GB uss=26.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=375.3MB pss=301.9MB uss=249.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.8GB uss=26.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=385.9MB pss=312.8MB uss=260.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=375.4MB pss=302.0MB uss=249.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.8GB uss=26.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=386.0MB pss=312.9MB uss=260.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=376.5MB pss=303.1MB uss=250.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.8GB uss=26.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=386.1MB pss=313.0MB uss=260.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.8GB uss=26.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=376.6MB pss=303.2MB uss=250.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.8GB uss=26.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=386.6MB pss=313.6MB uss=261.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.8GB uss=26.6GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=24.7GB pss=24.6GB uss=24.6GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=26.9GB uss=26.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=386.8MB pss=313.7MB uss=261.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=376.7MB pss=303.3MB uss=250.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.9GB uss=26.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=386.9MB pss=313.8MB uss=261.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=377.3MB pss=303.9MB uss=251.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.9GB uss=26.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=387.0MB pss=313.9MB uss=261.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=377.4MB pss=304.0MB uss=251.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=26.9GB uss=26.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=387.1MB pss=314.0MB uss=261.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=26.9GB uss=26.7GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=24.8GB pss=24.7GB uss=24.7GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=387.3MB pss=314.2MB uss=261.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=377.5MB pss=304.1MB uss=251.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.0GB uss=26.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=387.4MB pss=314.3MB uss=261.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.0GB uss=26.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=377.6MB pss=304.2MB uss=251.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.0GB uss=26.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=387.5MB pss=314.4MB uss=261.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.0GB uss=26.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=387.6MB pss=314.5MB uss=262.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=377.7MB pss=304.3MB uss=251.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.0GB uss=26.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=388.6MB pss=315.5MB uss=263.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=378.2MB pss=304.8MB uss=252.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.0GB uss=26.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=388.7MB pss=315.6MB uss=263.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.0GB uss=26.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=388.8MB pss=315.7MB uss=263.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.0GB uss=26.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=24.9GB pss=24.8GB uss=24.8GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=27.1GB uss=26.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=388.9MB pss=315.8MB uss=263.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.1GB uss=26.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=389.4MB pss=316.4MB uss=263.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.1GB uss=26.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=378.3MB pss=304.9MB uss=252.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.1GB uss=26.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=389.5MB pss=316.4MB uss=263.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.1GB uss=26.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=378.4MB pss=305.0MB uss=252.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.1GB uss=26.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=379.0MB pss=305.6MB uss=252.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.1GB uss=26.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=389.6MB pss=316.5MB uss=264.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.1GB uss=26.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=379.1MB pss=305.7MB uss=252.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.1GB uss=26.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=389.7MB pss=316.6MB uss=264.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=379.2MB pss=305.8MB uss=253.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.1GB uss=26.9GB | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=7.5GB, std_dev=0.0B, min=7.5GB, max=7.5GB, num_seqs=4203, avg_data_len=1) | |
| train epoch 69, finished after 4203 steps, 0:19:31 elapsed (99.4% computing time) | |
| Save model under /u/maximilian.kannen/setups/20230406_feat/work/i6_core/returnn/training/ReturnnTrainingJob.wQIA3Rc1nLak/output/models/epoch.069 | |
| MEMORY: main proc python3(1781720) increased RSS: rss=25.1GB pss=25.0GB uss=25.0GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=27.3GB uss=27.2GB | |
| epoch 69 score: 1.2418555095627608 error: None elapsed: 0:19:31 | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=7.5GB, std_dev=0.0B, min=7.5GB, max=7.5GB, num_seqs=31, avg_data_len=1) | |
| epoch 69 'dev' eval, finished after 31 steps, 0:00:03 elapsed (63.4% computing time) | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=389.8MB pss=316.7MB uss=264.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=379.3MB pss=305.9MB uss=253.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.3GB uss=27.2GB | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=7.5GB, std_dev=0.0B, min=7.5GB, max=7.5GB, num_seqs=29, avg_data_len=1) | |
| epoch 69 'devtrain' eval, finished after 29 steps, 0:00:03 elapsed (62.6% computing time) | |
| dev: score 1.4889442486616957 error None devtrain: score 1.4744862890409158 error None | |
| We have stored models for epochs [4, 5, 6, ..., 67, 68, 69] and keep epochs [4, 5, 6, 7, 8, 9, 10, 11, 65, 66, 67, 68, 69]. | |
| We will delete the models of epochs [64]. | |
| Deleted 335.5MB. | |
| start epoch 70 with learning rate 0.00016150416666666666 ... | |
| [2023-10-31 03:00:57,446] INFO: [32mRun time: 2:56:17 CPU: 0.00% RSS: 28.92GB VMS: 69.29GB[0m | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=379.4MB pss=306.0MB uss=253.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.3GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=389.9MB pss=316.8MB uss=264.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.3GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=390.0MB pss=316.9MB uss=264.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.3GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=379.5MB pss=306.1MB uss=253.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.3GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=390.2MB pss=317.1MB uss=264.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=379.6MB pss=306.2MB uss=253.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.3GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=390.3MB pss=317.2MB uss=264.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=379.7MB pss=306.3MB uss=253.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.3GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=390.4MB pss=317.3MB uss=264.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.3GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=390.5MB pss=317.4MB uss=264.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=379.9MB pss=306.5MB uss=253.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.3GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=390.6MB pss=317.5MB uss=265.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.3GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=380.0MB pss=306.6MB uss=253.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.3GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=390.7MB pss=317.6MB uss=265.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=380.5MB pss=307.1MB uss=254.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.3GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=391.7MB pss=318.6MB uss=266.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=381.0MB pss=307.6MB uss=254.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.3GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=391.8MB pss=318.7MB uss=266.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=381.1MB pss=307.7MB uss=254.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.3GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=391.9MB pss=318.8MB uss=266.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=381.2MB pss=307.8MB uss=255.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.3GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=392.0MB pss=318.9MB uss=266.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=381.3MB pss=307.9MB uss=255.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.3GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=392.6MB pss=319.5MB uss=267.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=381.4MB pss=308.0MB uss=255.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.3GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=381.5MB pss=308.1MB uss=255.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.3GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=381.6MB pss=308.2MB uss=255.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.3GB uss=27.2GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=25.2GB pss=25.1GB uss=25.1GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=27.4GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=382.0MB pss=308.7MB uss=255.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.4GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=392.7MB pss=319.6MB uss=267.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=382.1MB pss=308.7MB uss=255.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.4GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=392.8MB pss=319.7MB uss=267.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.4GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=392.9MB pss=319.8MB uss=267.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=382.2MB pss=308.8MB uss=256.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.4GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=393.0MB pss=319.9MB uss=267.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.4GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=393.1MB pss=320.0MB uss=267.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=382.4MB pss=309.0MB uss=256.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.4GB uss=27.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=393.2MB pss=320.1MB uss=267.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=382.5MB pss=309.1MB uss=256.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.4GB uss=27.2GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=25.3GB pss=25.2GB uss=25.2GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=27.5GB uss=27.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=382.6MB pss=309.2MB uss=256.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.5GB uss=27.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=393.3MB pss=320.2MB uss=267.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.5GB uss=27.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=393.4MB pss=320.3MB uss=267.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.5GB uss=27.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=393.5MB pss=320.4MB uss=267.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.5GB uss=27.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=382.7MB pss=309.3MB uss=256.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.5GB uss=27.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=393.6MB pss=320.5MB uss=268.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.5GB uss=27.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=394.2MB pss=321.1MB uss=268.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=382.8MB pss=309.4MB uss=256.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.5GB uss=27.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=394.3MB pss=321.2MB uss=268.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.5GB uss=27.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=394.4MB pss=321.3MB uss=268.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=383.0MB pss=309.6MB uss=256.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.5GB uss=27.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=383.5MB pss=310.2MB uss=257.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.5GB uss=27.3GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=25.4GB pss=25.3GB uss=25.3GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=383.6MB pss=310.2MB uss=257.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.6GB uss=27.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=394.5MB pss=321.4MB uss=268.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=383.8MB pss=310.4MB uss=257.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.6GB uss=27.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=383.9MB pss=310.5MB uss=257.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.6GB uss=27.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=394.6MB pss=321.5MB uss=269.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.6GB uss=27.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=394.7MB pss=321.6MB uss=269.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=384.0MB pss=310.6MB uss=257.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.6GB uss=27.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=394.8MB pss=321.7MB uss=269.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.6GB uss=27.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=395.0MB pss=321.9MB uss=269.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=384.1MB pss=310.7MB uss=257.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.6GB uss=27.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=395.1MB pss=322.0MB uss=269.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.6GB uss=27.4GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=25.5GB pss=25.4GB uss=25.4GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=395.7MB pss=322.7MB uss=270.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=384.2MB pss=310.8MB uss=258.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.7GB uss=27.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=396.2MB pss=323.2MB uss=270.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.7GB uss=27.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=384.3MB pss=310.9MB uss=258.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.7GB uss=27.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=396.3MB pss=323.2MB uss=270.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.7GB uss=27.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=384.4MB pss=311.1MB uss=258.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.7GB uss=27.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=385.4MB pss=312.1MB uss=259.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.7GB uss=27.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=396.4MB pss=323.3MB uss=270.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.7GB uss=27.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=385.5MB pss=312.1MB uss=259.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.7GB uss=27.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=396.5MB pss=323.4MB uss=270.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.7GB uss=27.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=396.6MB pss=323.6MB uss=271.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=385.6MB pss=312.2MB uss=259.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.7GB uss=27.5GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=25.6GB pss=25.5GB uss=25.5GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=396.8MB pss=323.7MB uss=271.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.8GB uss=27.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=385.7MB pss=312.3MB uss=259.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.8GB uss=27.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=396.9MB pss=323.8MB uss=271.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=385.8MB pss=312.4MB uss=259.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.8GB uss=27.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=385.9MB pss=312.5MB uss=259.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.8GB uss=27.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=397.0MB pss=323.9MB uss=271.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=386.5MB pss=313.1MB uss=260.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.8GB uss=27.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=397.1MB pss=324.1MB uss=271.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=386.6MB pss=313.2MB uss=260.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.8GB uss=27.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=397.2MB pss=324.2MB uss=271.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.8GB uss=27.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=397.3MB pss=324.2MB uss=271.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.8GB uss=27.6GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=25.7GB pss=25.6GB uss=25.6GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=27.9GB uss=27.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=397.5MB pss=324.4MB uss=271.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=386.7MB pss=313.3MB uss=260.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.9GB uss=27.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=398.0MB pss=325.0MB uss=272.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.9GB uss=27.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=398.1MB pss=325.0MB uss=272.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.9GB uss=27.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=386.8MB pss=313.4MB uss=260.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.9GB uss=27.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=398.3MB pss=325.2MB uss=272.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.9GB uss=27.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=398.4MB pss=325.3MB uss=272.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=386.9MB pss=313.5MB uss=260.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.9GB uss=27.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=398.5MB pss=325.4MB uss=272.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.9GB uss=27.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=387.0MB pss=313.6MB uss=260.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.9GB uss=27.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=398.6MB pss=325.5MB uss=273.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=27.9GB uss=27.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=387.1MB pss=313.7MB uss=260.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=27.9GB uss=27.7GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=25.8GB pss=25.7GB uss=25.7GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=28.0GB uss=27.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=398.7MB pss=325.6MB uss=273.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.0GB uss=27.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=387.2MB pss=313.8MB uss=261.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.0GB uss=27.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=398.8MB pss=325.7MB uss=273.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=387.3MB pss=314.0MB uss=261.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.0GB uss=27.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=398.9MB pss=325.8MB uss=273.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=387.4MB pss=314.0MB uss=261.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.0GB uss=27.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=399.6MB pss=326.5MB uss=274.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=388.1MB pss=314.7MB uss=261.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.0GB uss=27.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=399.7MB pss=326.6MB uss=274.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.0GB uss=27.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=399.9MB pss=326.8MB uss=274.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=388.2MB pss=314.8MB uss=262.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.0GB uss=27.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=25.9GB pss=25.8GB uss=25.8GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=28.1GB uss=28.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=400.0MB pss=326.9MB uss=274.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=388.3MB pss=314.9MB uss=262.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.1GB uss=28.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=400.1MB pss=327.0MB uss=274.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.1GB uss=28.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=388.5MB pss=315.1MB uss=262.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.1GB uss=28.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=400.2MB pss=327.1MB uss=274.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.1GB uss=28.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=388.6MB pss=315.2MB uss=262.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.1GB uss=28.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=400.4MB pss=327.3MB uss=274.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=388.8MB pss=315.4MB uss=262.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.1GB uss=28.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=400.5MB pss=327.4MB uss=274.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.1GB uss=28.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=388.9MB pss=315.5MB uss=262.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.1GB uss=28.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=400.6MB pss=327.5MB uss=275.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.1GB uss=28.0GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=26.0GB pss=25.9GB uss=25.9GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=401.6MB pss=328.5MB uss=276.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=389.5MB pss=316.2MB uss=263.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.2GB uss=28.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=389.6MB pss=316.2MB uss=263.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.2GB uss=28.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=389.7MB pss=316.3MB uss=263.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.2GB uss=28.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=401.7MB pss=328.6MB uss=276.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.2GB uss=28.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=389.8MB pss=316.4MB uss=263.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.2GB uss=28.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=401.8MB pss=328.8MB uss=276.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.2GB uss=28.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=401.9MB pss=328.8MB uss=276.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.2GB uss=28.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=389.9MB pss=316.5MB uss=263.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.2GB uss=28.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=402.0MB pss=328.9MB uss=276.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.2GB uss=28.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=402.2MB pss=329.1MB uss=276.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.2GB uss=28.1GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=26.1GB pss=26.0GB uss=26.0GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=402.6MB pss=329.6MB uss=277.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=390.0MB pss=316.6MB uss=263.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.3GB uss=28.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=402.7MB pss=329.6MB uss=277.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=390.1MB pss=316.7MB uss=263.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.3GB uss=28.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=390.2MB pss=316.8MB uss=264.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.3GB uss=28.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=402.8MB pss=329.7MB uss=277.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=390.3MB pss=316.9MB uss=264.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.3GB uss=28.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=390.4MB pss=317.0MB uss=264.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.3GB uss=28.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=391.0MB pss=317.6MB uss=264.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.3GB uss=28.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=402.9MB pss=329.8MB uss=277.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.3GB uss=28.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=391.1MB pss=317.7MB uss=264.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.3GB uss=28.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=403.0MB pss=329.9MB uss=277.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=391.6MB pss=318.2MB uss=265.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.3GB uss=28.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=403.1MB pss=330.0MB uss=277.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.3GB uss=28.2GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=26.2GB pss=26.1GB uss=26.1GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=403.2MB pss=330.1MB uss=277.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.4GB uss=28.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=403.3MB pss=330.2MB uss=277.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.4GB uss=28.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=403.4MB pss=330.3MB uss=277.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.4GB uss=28.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=404.1MB pss=331.0MB uss=278.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=391.7MB pss=318.3MB uss=265.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.4GB uss=28.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=404.2MB pss=331.1MB uss=278.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=391.8MB pss=318.4MB uss=265.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.4GB uss=28.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=404.4MB pss=331.3MB uss=278.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=391.9MB pss=318.6MB uss=265.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.4GB uss=28.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=404.5MB pss=331.4MB uss=278.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=392.5MB pss=319.1MB uss=266.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.4GB uss=28.3GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=26.3GB pss=26.2GB uss=26.2GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=404.6MB pss=331.5MB uss=279.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=392.6MB pss=319.3MB uss=266.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.5GB uss=28.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=392.7MB pss=319.3MB uss=266.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.5GB uss=28.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=404.7MB pss=331.6MB uss=279.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.5GB uss=28.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=392.8MB pss=319.4MB uss=266.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.5GB uss=28.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=404.9MB pss=331.8MB uss=279.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.5GB uss=28.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=405.0MB pss=331.9MB uss=279.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=392.9MB pss=319.5MB uss=266.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.5GB uss=28.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=405.7MB pss=332.6MB uss=280.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=393.1MB pss=319.7MB uss=266.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.5GB uss=28.4GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=26.4GB pss=26.3GB uss=26.3GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=405.8MB pss=332.7MB uss=280.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=393.2MB pss=319.8MB uss=267.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.6GB uss=28.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=405.9MB pss=332.8MB uss=280.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=393.3MB pss=319.9MB uss=267.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.6GB uss=28.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=406.0MB pss=332.9MB uss=280.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=393.4MB pss=320.0MB uss=267.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.6GB uss=28.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=393.5MB pss=320.1MB uss=267.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.6GB uss=28.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=393.9MB pss=320.6MB uss=267.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.6GB uss=28.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=394.0MB pss=320.7MB uss=267.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.6GB uss=28.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=406.1MB pss=333.0MB uss=280.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=394.1MB pss=320.7MB uss=267.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.6GB uss=28.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=406.2MB pss=333.1MB uss=280.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.6GB uss=28.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=394.2MB pss=320.8MB uss=268.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.6GB uss=28.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=406.3MB pss=333.2MB uss=280.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.6GB uss=28.5GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=26.5GB pss=26.4GB uss=26.4GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=406.4MB pss=333.4MB uss=280.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.7GB uss=28.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=406.5MB pss=333.5MB uss=281.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=394.3MB pss=320.9MB uss=268.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.7GB uss=28.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=407.1MB pss=334.0MB uss=281.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.7GB uss=28.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=394.4MB pss=321.0MB uss=268.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.7GB uss=28.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=407.2MB pss=334.1MB uss=281.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=394.5MB pss=321.1MB uss=268.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.7GB uss=28.6GB | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=7.5GB, std_dev=0.0B, min=7.5GB, max=7.5GB, num_seqs=4223, avg_data_len=1) | |
| train epoch 70, finished after 4223 steps, 0:19:27 elapsed (99.4% computing time) | |
| Save model under /u/maximilian.kannen/setups/20230406_feat/work/i6_core/returnn/training/ReturnnTrainingJob.wQIA3Rc1nLak/output/models/epoch.070 | |
| MEMORY: main proc python3(1781720) increased RSS: rss=26.7GB pss=26.7GB uss=26.6GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| epoch 70 score: 1.242224259183981 error: None elapsed: 0:19:27 | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=7.5GB, std_dev=0.0B, min=7.5GB, max=7.5GB, num_seqs=31, avg_data_len=1) | |
| epoch 70 'dev' eval, finished after 31 steps, 0:00:03 elapsed (63.3% computing time) | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=407.3MB pss=334.2MB uss=281.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=394.7MB pss=321.3MB uss=268.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=7.5GB, std_dev=0.0B, min=7.5GB, max=7.5GB, num_seqs=29, avg_data_len=1) | |
| epoch 70 'devtrain' eval, finished after 29 steps, 0:00:03 elapsed (60.9% computing time) | |
| dev: score 1.3919418921139897 error None devtrain: score 1.3796773516088352 error None | |
| We have stored models for epochs [4, 5, 6, ..., 68, 69, 70] and keep epochs [4, 5, 6, 7, 8, 9, 10, 11, 66, 67, 68, 69, 70]. | |
| We will delete the models of epochs [65]. | |
| Deleted 335.5MB. | |
| start epoch 71 with learning rate 0.00016365277777777778 ... | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=394.8MB pss=321.4MB uss=268.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=394.9MB pss=321.5MB uss=268.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=407.4MB pss=334.3MB uss=281.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=395.9MB pss=322.5MB uss=269.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=407.5MB pss=334.4MB uss=281.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=396.0MB pss=322.6MB uss=269.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=407.7MB pss=334.6MB uss=282.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=407.8MB pss=334.7MB uss=282.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=396.1MB pss=322.7MB uss=269.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=407.9MB pss=334.8MB uss=282.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=408.1MB pss=335.0MB uss=282.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=408.6MB pss=335.5MB uss=283.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=396.2MB pss=322.8MB uss=270.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=408.7MB pss=335.6MB uss=283.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=396.3MB pss=322.9MB uss=270.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=408.8MB pss=335.7MB uss=283.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=396.8MB pss=323.4MB uss=270.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=408.9MB pss=335.8MB uss=283.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=396.9MB pss=323.5MB uss=270.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=409.0MB pss=336.0MB uss=283.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=397.0MB pss=323.6MB uss=270.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=409.1MB pss=336.0MB uss=283.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=397.1MB pss=323.7MB uss=270.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=409.2MB pss=336.1MB uss=283.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=397.2MB pss=323.8MB uss=271.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=409.3MB pss=336.3MB uss=283.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=397.3MB pss=323.9MB uss=271.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=409.4MB pss=336.3MB uss=283.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=409.5MB pss=336.5MB uss=284.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=397.5MB pss=324.1MB uss=271.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=410.6MB pss=337.5MB uss=285.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=397.6MB pss=324.2MB uss=271.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=28.9GB uss=28.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=26.8GB pss=26.7GB uss=26.7GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=29.0GB uss=28.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=398.2MB pss=324.8MB uss=272.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.0GB uss=28.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=410.7MB pss=337.6MB uss=285.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.0GB uss=28.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=398.3MB pss=324.9MB uss=272.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.0GB uss=28.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=410.8MB pss=337.7MB uss=285.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.0GB uss=28.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=410.9MB pss=337.8MB uss=285.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.0GB uss=28.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=411.4MB pss=338.3MB uss=285.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=398.4MB pss=325.0MB uss=272.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.0GB uss=28.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=398.5MB pss=325.1MB uss=272.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.0GB uss=28.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=411.5MB pss=338.4MB uss=285.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.0GB uss=28.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=398.6MB pss=325.2MB uss=272.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.0GB uss=28.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=411.6MB pss=338.5MB uss=286.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.0GB uss=28.9GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=412.1MB pss=339.0MB uss=286.5MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=398.7MB pss=325.3MB uss=272.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.0GB uss=28.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=26.9GB pss=26.8GB uss=26.8GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=29.1GB uss=29.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=412.2MB pss=339.1MB uss=286.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=398.8MB pss=325.4MB uss=272.6MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.1GB uss=29.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=398.9MB pss=325.5MB uss=272.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.1GB uss=29.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=412.3MB pss=339.2MB uss=286.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=399.1MB pss=325.7MB uss=272.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.1GB uss=29.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=412.4MB pss=339.3MB uss=286.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.1GB uss=29.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=399.2MB pss=325.8MB uss=273.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.1GB uss=29.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=412.5MB pss=339.4MB uss=286.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.1GB uss=29.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=412.6MB pss=339.5MB uss=287.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=399.4MB pss=326.0MB uss=273.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.1GB uss=29.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=399.5MB pss=326.1MB uss=273.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.1GB uss=29.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=400.0MB pss=326.6MB uss=273.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.1GB uss=29.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=412.7MB pss=339.6MB uss=287.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=400.1MB pss=326.8MB uss=274.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.1GB uss=29.0GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=400.2MB pss=326.8MB uss=274.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.1GB uss=29.0GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=27.0GB pss=26.9GB uss=26.9GB shared=49.3MB | |
| MEMORY: total (7 procs): pss=29.2GB uss=29.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=412.8MB pss=339.7MB uss=287.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.2GB uss=29.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=412.9MB pss=339.8MB uss=287.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=400.3MB pss=326.9MB uss=274.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.2GB uss=29.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=413.0MB pss=339.9MB uss=287.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.2GB uss=29.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=413.1MB pss=340.0MB uss=287.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.2GB uss=29.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=413.2MB pss=340.1MB uss=287.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=400.4MB pss=327.0MB uss=274.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.2GB uss=29.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=413.3MB pss=340.2MB uss=287.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.2GB uss=29.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=413.9MB pss=340.8MB uss=288.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=400.5MB pss=327.1MB uss=274.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.2GB uss=29.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=414.0MB pss=340.9MB uss=288.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=400.6MB pss=327.2MB uss=274.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.2GB uss=29.1GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=414.1MB pss=341.0MB uss=288.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.2GB uss=29.1GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=27.1GB pss=27.0GB uss=27.0GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=400.8MB pss=327.4MB uss=274.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.3GB uss=29.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=400.9MB pss=327.5MB uss=274.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.3GB uss=29.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=414.2MB pss=341.1MB uss=288.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=401.4MB pss=328.0MB uss=275.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.3GB uss=29.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=401.8MB pss=328.4MB uss=275.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.3GB uss=29.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=401.9MB pss=328.5MB uss=275.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.3GB uss=29.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=414.3MB pss=341.2MB uss=288.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.3GB uss=29.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=414.4MB pss=341.4MB uss=288.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=402.0MB pss=328.6MB uss=275.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.3GB uss=29.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=414.5MB pss=341.4MB uss=288.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.3GB uss=29.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=414.6MB pss=341.5MB uss=289.0MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.3GB uss=29.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=414.8MB pss=341.7MB uss=289.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=402.1MB pss=328.8MB uss=276.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.3GB uss=29.2GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=415.3MB pss=342.3MB uss=289.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=402.2MB pss=328.8MB uss=276.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.3GB uss=29.2GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=27.2GB pss=27.1GB uss=27.1GB shared=49.3MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=415.4MB pss=342.3MB uss=289.8MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.4GB uss=29.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=415.5MB pss=342.4MB uss=289.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.4GB uss=29.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=402.3MB pss=328.9MB uss=276.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.4GB uss=29.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=402.4MB pss=329.0MB uss=276.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.4GB uss=29.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=415.6MB pss=342.5MB uss=290.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=402.9MB pss=329.5MB uss=276.7MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.4GB uss=29.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=415.7MB pss=342.6MB uss=290.1MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=403.4MB pss=330.0MB uss=277.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.4GB uss=29.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=415.9MB pss=342.8MB uss=290.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=403.5MB pss=330.1MB uss=277.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.4GB uss=29.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=416.0MB pss=342.9MB uss=290.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.4GB uss=29.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=416.1MB pss=343.0MB uss=290.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.4GB uss=29.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=416.7MB pss=343.7MB uss=291.2MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=403.6MB pss=330.2MB uss=277.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.4GB uss=29.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=416.8MB pss=343.7MB uss=291.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=403.7MB pss=330.3MB uss=277.5MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.4GB uss=29.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=417.0MB pss=343.9MB uss=291.4MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=404.2MB pss=330.8MB uss=278.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.4GB uss=29.3GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=417.1MB pss=344.0MB uss=291.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.4GB uss=29.3GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=27.3GB pss=27.2GB uss=27.2GB shared=46.9MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=417.2MB pss=344.1MB uss=291.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.5GB uss=29.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=404.3MB pss=330.9MB uss=278.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.5GB uss=29.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=417.3MB pss=344.2MB uss=291.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.5GB uss=29.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=417.4MB pss=344.3MB uss=291.8MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=404.4MB pss=331.0MB uss=278.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.5GB uss=29.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=404.5MB pss=331.1MB uss=278.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.5GB uss=29.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=404.6MB pss=331.2MB uss=278.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.5GB uss=29.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=417.5MB pss=344.5MB uss=292.0MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=405.1MB pss=331.7MB uss=278.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.5GB uss=29.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=417.6MB pss=344.5MB uss=292.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.5GB uss=29.4GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=417.7MB pss=344.6MB uss=292.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.5GB uss=29.4GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=27.4GB pss=27.3GB uss=27.3GB shared=46.9MB | |
| MEMORY: total (7 procs): pss=29.6GB uss=29.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=417.8MB pss=344.8MB uss=292.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.6GB uss=29.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=417.9MB pss=344.8MB uss=292.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.6GB uss=29.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=418.8MB pss=345.8MB uss=293.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.6GB uss=29.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=418.9MB pss=345.8MB uss=293.3MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.6GB uss=29.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=419.0MB pss=345.9MB uss=293.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.6GB uss=29.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=405.3MB pss=331.9MB uss=279.1MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.6GB uss=29.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=419.1MB pss=346.0MB uss=293.5MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.6GB uss=29.5GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=405.4MB pss=332.0MB uss=279.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.6GB uss=29.5GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=27.5GB pss=27.4GB uss=27.4GB shared=46.9MB | |
| MEMORY: total (7 procs): pss=29.7GB uss=29.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=419.2MB pss=346.1MB uss=293.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.7GB uss=29.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=405.5MB pss=332.1MB uss=279.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.7GB uss=29.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=419.3MB pss=346.2MB uss=293.7MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.7GB uss=29.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=405.6MB pss=332.2MB uss=279.4MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.7GB uss=29.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=419.9MB pss=346.8MB uss=294.3MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=406.1MB pss=332.7MB uss=279.9MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.7GB uss=29.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=420.0MB pss=346.9MB uss=294.4MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.7GB uss=29.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=420.1MB pss=347.1MB uss=294.6MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=406.2MB pss=332.8MB uss=280.0MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.7GB uss=29.6GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=420.2MB pss=347.1MB uss=294.6MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.7GB uss=29.6GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=27.6GB pss=27.5GB uss=27.5GB shared=46.9MB | |
| MEMORY: total (7 procs): pss=29.8GB uss=29.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=406.3MB pss=333.0MB uss=280.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.8GB uss=29.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=420.3MB pss=347.2MB uss=294.7MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=406.4MB pss=333.0MB uss=280.2MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.8GB uss=29.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=406.5MB pss=333.1MB uss=280.3MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.8GB uss=29.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=420.4MB pss=347.4MB uss=294.9MB shared=125.6MB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783214) increased RSS: rss=407.0MB pss=333.6MB uss=280.8MB shared=126.2MB | |
| MEMORY: total (7 procs): pss=29.8GB uss=29.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=420.5MB pss=347.4MB uss=294.9MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.8GB uss=29.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=420.7MB pss=347.6MB uss=295.1MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.8GB uss=29.7GB | |
| MEMORY: sub proc nn-trainer.linux-x86_64-standard(1783196) increased RSS: rss=420.8MB pss=347.7MB uss=295.2MB shared=125.6MB | |
| MEMORY: total (7 procs): pss=29.8GB uss=29.7GB | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=7.5GB, std_dev=0.0B, min=7.5GB, max=7.5GB, num_seqs=4175, avg_data_len=1) | |
| train epoch 71, finished after 4175 steps, 0:18:55 elapsed (99.4% computing time) | |
| Save model under /u/maximilian.kannen/setups/20230406_feat/work/i6_core/returnn/training/ReturnnTrainingJob.wQIA3Rc1nLak/output/models/epoch.071 | |
| epoch 71 score: 1.2392126500554386 error: None elapsed: 0:18:55 | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=7.5GB, std_dev=0.0B, min=7.5GB, max=7.5GB, num_seqs=31, avg_data_len=1) | |
| epoch 71 'dev' eval, finished after 31 steps, 0:00:03 elapsed (62.7% computing time) | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=7.5GB, std_dev=0.0B, min=7.5GB, max=7.5GB, num_seqs=29, avg_data_len=1) | |
| epoch 71 'devtrain' eval, finished after 29 steps, 0:00:03 elapsed (61.7% computing time) | |
| dev: score 1.3880622582619946 error None devtrain: score 1.376905326738325 error None | |
| We have stored models for epochs [4, 5, 6, ..., 69, 70, 71] and keep epochs [4, 5, 6, 7, 8, 9, 10, 11, 67, 68, 69, 70, 71]. | |
| We will delete the models of epochs [66]. | |
| Deleted 335.5MB. | |
| start epoch 72 with learning rate 0.00016580138888888887 ... | |
| MEMORY: main proc python3(1781720) increased RSS: rss=27.7GB pss=27.6GB uss=27.6GB shared=16.1MB | |
| MEMORY: total (7 procs): pss=29.9GB uss=29.8GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=27.8GB pss=27.7GB uss=27.7GB shared=12.9MB | |
| MEMORY: total (7 procs): pss=30.0GB uss=29.9GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=27.9GB pss=27.8GB uss=27.8GB shared=12.9MB | |
| MEMORY: total (7 procs): pss=30.1GB uss=30.0GB | |
| MEMORY: main proc python3(1781720) increased RSS: rss=28.0GB pss=27.9GB uss=27.9GB shared=12.9MB | |
| MEMORY: total (7 procs): pss=30.2GB uss=30.1GB | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=7.5GB, std_dev=0.0B, min=7.5GB, max=7.5GB, num_seqs=4184, avg_data_len=1) | |
| train epoch 72, finished after 4184 steps, 0:18:51 elapsed (99.4% computing time) | |
| Save model under /u/maximilian.kannen/setups/20230406_feat/work/i6_core/returnn/training/ReturnnTrainingJob.wQIA3Rc1nLak/output/models/epoch.072 | |
| epoch 72 score: 1.2398413123197733 error: None elapsed: 0:18:51 | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=7.5GB, std_dev=0.0B, min=7.5GB, max=7.5GB, num_seqs=31, avg_data_len=1) | |
| epoch 72 'dev' eval, finished after 31 steps, 0:00:03 elapsed (63.8% computing time) | |
| Stats: | |
| mem_usage:GPU:0: Stats(mean=7.5GB, std_dev=0.0B, min=7.5GB, max=7.5GB, num_seqs=29, avg_data_len=1) | |
| epoch 72 'devtrain' eval, finished after 29 steps, 0:00:03 elapsed (62.0% computing time) | |
| dev: score 1.3886432499783739 error None devtrain: score 1.378037299632041 error None | |
| We have stored models for epochs [4, 5, 6, ..., 70, 71, 72] and keep epochs [4, 5, 6, 7, 8, 9, 10, 11, 68, 69, 70, 71, 72]. | |
| We will delete the models of epochs [67]. | |
| Deleted 335.5MB. | |
| start epoch 73 with learning rate 0.00016795 ... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment