(partial) EasyBuild log for failed build of /home/boegelbot/easybuild/easybuild-easyconfigs/easybuild/easyconfigs/p/PyTorch/PyTorch-1.8.1-fosscuda-2019b-Python-3.7.4.eb (PR(s) #13795)

PyTorch-1.8.1-fosscuda-2019b-Python-3.7.4_partial.log

distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3] SKIPPED [  7%]
distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2] SKIPPED [ 15%]
distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1] SKIPPED [ 23%]
distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1] SKIPPED [ 30%]
distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3] SKIPPED [ 38%]
distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2] SKIPPED [ 46%]
distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1] SKIPPED [ 53%]
distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1] SKIPPED [ 61%]
distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3] SKIPPED [ 69%]
distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2] SKIPPED [ 76%]
distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1] SKIPPED [ 84%]
distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1] SKIPPED [ 92%]
distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip PASSED      [100%]

======================== 1 passed, 12 skipped in 0.10s =========================
Running distributed/pipeline/sync/skip/test_inspect_skip_layout ... [2021-09-20 19:59:38.578246]
Executing ['/project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_inspect_skip_layout.py', '-v'] ... [2021-09-20 19:59:38.578406]
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.13.0 -- /project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch/test/.hypothesis/examples')
torch: 1.8.1
rootdir: /tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch
plugins: hypothesis-4.44.2
collecting ... collected 6 items

distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables PASSED [ 16%]
distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition PASSED [ 33%]
distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions PASSED [ 50%]
distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions PASSED [ 66%]
distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions PASSED [ 83%]
distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace PASSED [100%]

============================== 6 passed in 0.05s ===============================
Running distributed/pipeline/sync/skip/test_leak ... [2021-09-20 19:59:39.852423]
Executing ['/project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_leak.py', '-v'] ... [2021-09-20 19:59:39.852564]
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.13.0 -- /project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch/test/.hypothesis/examples')
torch: 1.8.1
rootdir: /tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch
plugins: hypothesis-4.44.2
collecting ... collected 8 items

distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train] PASSED [ 12%]
distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval] PASSED [ 25%]
distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train] PASSED [ 37%]
distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval] PASSED [ 50%]
distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train] PASSED [ 62%]
distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval] PASSED [ 75%]
distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train] PASSED [ 87%]
distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval] PASSED [100%]

============================== 8 passed in 0.28s ===============================
Running distributed/pipeline/sync/skip/test_portal ... [2021-09-20 19:59:41.372037]
Executing ['/project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_portal.py', '-v'] ... [2021-09-20 19:59:41.372181]
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.13.0 -- /project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch/test/.hypothesis/examples')
torch: 1.8.1
rootdir: /tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch
plugins: hypothesis-4.44.2
collecting ... collected 10 items

distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device SKIPPED [ 10%]
distributed/pipeline/sync/skip/test_portal.py::test_blue_orange PASSED   [ 20%]
distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad PASSED [ 30%]
distributed/pipeline/sync/skip/test_portal.py::test_use_grad PASSED      [ 40%]
distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0 PASSED [ 50%]
distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1 PASSED [ 60%]
distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2 PASSED [ 70%]
distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3 PASSED [ 80%]
distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4 PASSED [ 90%]
distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 PASSED [100%]

========================= 9 passed, 1 skipped in 0.08s =========================
Running distributed/pipeline/sync/skip/test_stash_pop ... [2021-09-20 19:59:42.680727]
Executing ['/project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_stash_pop.py', '-v'] ... [2021-09-20 19:59:42.680858]
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.13.0 -- /project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch/test/.hypothesis/examples')
torch: 1.8.1
rootdir: /tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch
plugins: hypothesis-4.44.2
collecting ... collected 7 items

distributed/pipeline/sync/skip/test_stash_pop.py::test_stash PASSED      [ 14%]
distributed/pipeline/sync/skip/test_stash_pop.py::test_pop PASSED        [ 28%]
distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use PASSED [ 42%]
distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared PASSED [ 57%]
distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared PASSED [ 71%]
distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed PASSED [ 85%]
distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none PASSED [100%]

============================== 7 passed in 0.05s ===============================
Running distributed/pipeline/sync/skip/test_tracker ... [2021-09-20 19:59:43.889146]
Executing ['/project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_tracker.py', '-v'] ... [2021-09-20 19:59:43.889266]
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.13.0 -- /project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch/test/.hypothesis/examples')
torch: 1.8.1
rootdir: /tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch
plugins: hypothesis-4.44.2
collecting ... collected 6 items

distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker PASSED [ 16%]
distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel SKIPPED [ 33%]
distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal PASSED [ 50%]
distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal PASSED [ 66%]
distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing PASSED [ 83%]
distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing PASSED [100%]

========================= 5 passed, 1 skipped in 0.07s =========================
Running distributed/pipeline/sync/skip/test_verify_skippables ... [2021-09-20 19:59:45.220096]
Executing ['/project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_verify_skippables.py', '-v'] ... [2021-09-20 19:59:45.220228]
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.13.0 -- /project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch/test/.hypothesis/examples')
torch: 1.8.1
rootdir: /tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch
plugins: hypothesis-4.44.2
collecting ... collected 9 items

distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching PASSED [ 11%]
distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop PASSED [ 22%]
distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown PASSED [ 33%]
distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again PASSED [ 44%]
distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again PASSED [ 55%]
distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names PASSED [ 66%]
distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name PASSED [ 77%]
distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop PASSED [ 88%]
distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated PASSED [100%]

============================== 9 passed in 0.06s ===============================
Running distributed/pipeline/sync/test_balance ... [2021-09-20 19:59:46.460213]
Executing ['/project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_balance.py', '-v'] ... [2021-09-20 19:59:46.460363]
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.13.0 -- /project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch/test/.hypothesis/examples')
torch: 1.8.1
rootdir: /tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch
plugins: hypothesis-4.44.2
collecting ... collected 15 items

distributed/pipeline/sync/test_balance.py::test_blockpartition PASSED    [  6%]
distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros PASSED [ 13%]
distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions PASSED [ 20%]
distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence PASSED [ 26%]
distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu] SKIPPED [ 33%]
distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input PASSED [ 40%]
distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent SKIPPED [ 46%]
distributed/pipeline/sync/test_balance.py::test_balance_by_size_param SKIPPED [ 53%]
distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale SKIPPED [ 60%]
distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu] PASSED [ 66%]
distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu] PASSED [ 73%]
distributed/pipeline/sync/test_balance.py::test_not_training PASSED      [ 80%]
distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple PASSED [ 86%]
distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple SKIPPED [ 93%]
distributed/pipeline/sync/test_balance.py::test_already_has_grad PASSED  [100%]

======================== 10 passed, 5 skipped in 4.10s =========================
Running distributed/pipeline/sync/test_bugs ... [2021-09-20 19:59:51.762026]
Executing ['/project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_bugs.py', '-v'] ... [2021-09-20 19:59:51.762145]
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.13.0 -- /project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch/test/.hypothesis/examples')
torch: 1.8.1
rootdir: /tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch
plugins: hypothesis-4.44.2
collecting ... collected 4 items

distributed/pipeline/sync/test_bugs.py::test_python_autograd_function PASSED [ 25%]
distributed/pipeline/sync/test_bugs.py::test_exception_no_hang PASSED    [ 50%]
distributed/pipeline/sync/test_bugs.py::test_tuple_wait SKIPPED          [ 75%]
distributed/pipeline/sync/test_bugs.py::test_parallel_randoms PASSED     [100%]

========================= 3 passed, 1 skipped in 0.20s =========================
Running distributed/pipeline/sync/test_checkpoint ... [2021-09-20 19:59:53.336903]
Executing ['/project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_checkpoint.py', '-v'] ... [2021-09-20 19:59:53.337044]
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.13.0 -- /project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch/test/.hypothesis/examples')
torch: 1.8.1
rootdir: /tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch
plugins: hypothesis-4.44.2
collecting ... collected 7 items

distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu] PASSED [ 14%]
distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad PASSED [ 28%]
distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter PASSED [ 42%]
distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu] PASSED [ 57%]
distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing PASSED [ 71%]
distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint PASSED [ 85%]
distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output PASSED [100%]

============================== 7 passed in 0.06s ===============================
Running distributed/pipeline/sync/test_copy ... [2021-09-20 19:59:54.517025]
Executing ['/project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_copy.py', '-v'] ... [2021-09-20 19:59:54.517172]
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.13.0 -- /project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch/test/.hypothesis/examples')
torch: 1.8.1
rootdir: /tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch
plugins: hypothesis-4.44.2
collecting ... collected 5 items

distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu PASSED    [ 20%]
distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda SKIPPED  [ 40%]
distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu SKIPPED  [ 60%]
distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda SKIPPED [ 80%]
distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors PASSED [100%]

========================= 2 passed, 3 skipped in 0.04s =========================
Running distributed/pipeline/sync/test_deferred_batch_norm ... [2021-09-20 19:59:55.757789]
Executing ['/project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_deferred_batch_norm.py', '-v'] ... [2021-09-20 19:59:55.757924]
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.13.0 -- /project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch/test/.hypothesis/examples')
torch: 1.8.1
rootdir: /tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch
plugins: hypothesis-4.44.2
collecting ... collected 11 items

distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1] PASSED [  9%]
distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4] PASSED [ 18%]
distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1] PASSED [ 27%]
distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4] PASSED [ 36%]
distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1] PASSED [ 45%]
distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None] PASSED [ 54%]
distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm PASSED [ 63%]
distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval PASSED  [ 72%]
distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize PASSED [ 81%]
distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn PASSED [ 90%]
distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad PASSED [100%]

============================== 11 passed in 0.86s ==============================
Running distributed/pipeline/sync/test_dependency ... [2021-09-20 19:59:57.925751]
Executing ['/project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_dependency.py', '-v'] ... [2021-09-20 19:59:57.925877]
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.13.0 -- /project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch/test/.hypothesis/examples')
torch: 1.8.1
rootdir: /tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch
plugins: hypothesis-4.44.2
collecting ... collected 6 items

distributed/pipeline/sync/test_dependency.py::test_fork_join SKIPPED     [ 16%]
distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad PASSED [ 33%]
distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad PASSED [ 50%]
distributed/pipeline/sync/test_dependency.py::test_fork_leak PASSED      [ 66%]
distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad PASSED [ 83%]
distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad PASSED [100%]

========================= 5 passed, 1 skipped in 0.06s =========================
Running distributed/pipeline/sync/test_inplace ... [2021-09-20 19:59:59.175913]
Executing ['/project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_inplace.py', '-v'] ... [2021-09-20 19:59:59.176051]
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.13.0 -- /project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch/test/.hypothesis/examples')
torch: 1.8.1
rootdir: /tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch
plugins: hypothesis-4.44.2
collecting ... collected 3 items

distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad PASSED [ 33%]
distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad XFAIL [ 66%]
distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad XFAIL [100%]

========================= 1 passed, 2 xfailed in 0.18s =========================
Running distributed/pipeline/sync/test_microbatch ... [2021-09-20 20:00:00.681117]
Executing ['/project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_microbatch.py', '-v'] ... [2021-09-20 20:00:00.681238]
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.13.0 -- /project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch/test/.hypothesis/examples')
torch: 1.8.1
rootdir: /tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch
plugins: hypothesis-4.44.2
collecting ... collected 10 items

distributed/pipeline/sync/test_microbatch.py::test_batch_atomic PASSED   [ 10%]
distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic PASSED [ 20%]
distributed/pipeline/sync/test_microbatch.py::test_batch_call PASSED     [ 30%]
distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index PASSED [ 40%]
distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice PASSED [ 50%]
distributed/pipeline/sync/test_microbatch.py::test_check PASSED          [ 60%]
distributed/pipeline/sync/test_microbatch.py::test_gather_tensors PASSED [ 70%]
distributed/pipeline/sync/test_microbatch.py::test_gather_tuples PASSED  [ 80%]
distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor PASSED [ 90%]
distributed/pipeline/sync/test_microbatch.py::test_scatter_tuple PASSED  [100%]

============================== 10 passed in 0.07s ==============================
Running distributed/pipeline/sync/test_phony ... [2021-09-20 20:00:01.929657]
Executing ['/project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_phony.py', '-v'] ... [2021-09-20 20:00:01.929785]
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.13.0 -- /project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch/test/.hypothesis/examples')
torch: 1.8.1
rootdir: /tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch
plugins: hypothesis-4.44.2
collecting ... collected 4 items

distributed/pipeline/sync/test_phony.py::test_phony_size PASSED          [ 25%]
distributed/pipeline/sync/test_phony.py::test_phony_requires_grad PASSED [ 50%]
distributed/pipeline/sync/test_phony.py::test_cached_phony PASSED        [ 75%]
distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function PASSED [100%]

============================== 4 passed in 0.04s ===============================
Running distributed/pipeline/sync/test_pipe ... [2021-09-20 20:00:03.093946]
Executing ['/project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_pipe.py', '-v'] ... [2021-09-20 20:00:03.094077]
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.13.0 -- /project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch/test/.hypothesis/examples')
torch: 1.8.1
rootdir: /tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch
plugins: hypothesis-4.44.2
collecting ... collected 36 items

distributed/pipeline/sync/test_pipe.py::test_parameters PASSED           [  2%]
distributed/pipeline/sync/test_pipe.py::test_public_attrs PASSED         [  5%]
distributed/pipeline/sync/test_pipe.py::test_sequential_like PASSED      [  8%]
distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1 PASSED   [ 11%]
distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible PASSED [ 13%]
distributed/pipeline/sync/test_pipe.py::test_batch_size_small PASSED     [ 16%]
distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode PASSED      [ 19%]
distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid PASSED [ 22%]
distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1 PASSED [ 25%]
distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval PASSED      [ 27%]
distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input PASSED [ 30%]
distributed/pipeline/sync/test_pipe.py::test_no_grad PASSED              [ 33%]
distributed/pipeline/sync/test_pipe.py::test_exception PASSED            [ 36%]
distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap PASSED [ 38%]
distributed/pipeline/sync/test_pipe.py::test_nested_input PASSED         [ 41%]
distributed/pipeline/sync/test_pipe.py::test_input_pair PASSED           [ 44%]
distributed/pipeline/sync/test_pipe.py::test_input_singleton PASSED      [ 47%]
distributed/pipeline/sync/test_pipe.py::test_input_varargs PASSED        [ 50%]
distributed/pipeline/sync/test_pipe.py::test_non_tensor PASSED           [ 52%]
distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence PASSED  [ 55%]
distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never] PASSED [ 58%]
distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always] PASSED [ 61%]
distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last] PASSED [ 63%]
distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never] PASSED [ 66%]
distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always] PASSED [ 69%]
distributed/pipeline/sync/test_pipe.py::test_devices PASSED              [ 72%]
distributed/pipeline/sync/test_pipe.py::test_partitions PASSED           [ 75%]
distributed/pipeline/sync/test_pipe.py::test_deny_moving PASSED          [ 77%]
distributed/pipeline/sync/test_pipe.py::test_empty_module PASSED         [ 80%]
distributed/pipeline/sync/test_pipe.py::test_named_children PASSED       [ 83%]
distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential PASSED [ 86%]
distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children PASSED [ 88%]
distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device SKIPPED [ 91%]
distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules SKIPPED [ 94%]
distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device PASSED [ 97%]
distributed/pipeline/sync/test_pipe.py::test_forward_lockstep PASSED     [100%]

======================== 34 passed, 2 skipped in 1.14s =========================
Running distributed/pipeline/sync/test_pipeline ... [2021-09-20 20:00:05.546589]
Executing ['/project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_pipeline.py', '-v'] ... [2021-09-20 20:00:05.546749]
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.13.0 -- /project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch/test/.hypothesis/examples')
torch: 1.8.1
rootdir: /tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch
plugins: hypothesis-4.44.2
collecting ... collected 1 item

distributed/pipeline/sync/test_pipeline.py::test_clock_cycles PASSED     [100%]

============================== 1 passed in 0.03s ===============================
Running distributed/pipeline/sync/test_stream ... [2021-09-20 20:00:06.726101]
Executing ['/project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_stream.py', '-v'] ... [2021-09-20 20:00:06.726240]
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.13.0 -- /project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch/test/.hypothesis/examples')
torch: 1.8.1
rootdir: /tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch
plugins: hypothesis-4.44.2
collecting ... collected 19 items

distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu PASSED [  5%]
distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda SKIPPED [ 10%]
distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu PASSED [ 15%]
distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda SKIPPED [ 21%]
distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu PASSED [ 26%]
distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda SKIPPED [ 31%]
distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu PASSED [ 36%]
distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda SKIPPED [ 42%]
distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu PASSED [ 47%]
distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda SKIPPED [ 52%]
distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu PASSED [ 57%]
distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda SKIPPED [ 63%]
distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu PASSED [ 68%]
distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda SKIPPED [ 73%]
distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu SKIPPED [ 78%]
distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda SKIPPED [ 84%]
distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu PASSED [ 89%]
distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda SKIPPED [ 94%]
distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view SKIPPED [100%]

======================== 8 passed, 11 skipped in 0.09s =========================
Running distributed/pipeline/sync/test_transparency ... [2021-09-20 20:00:08.149399]
Executing ['/project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_transparency.py', '-v'] ... [2021-09-20 20:00:08.149550]
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.13.0 -- /project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch/test/.hypothesis/examples')
torch: 1.8.1
rootdir: /tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch
plugins: hypothesis-4.44.2
collecting ... collected 1 item

distributed/pipeline/sync/test_transparency.py::test_simple_linears PASSED [100%]

============================== 1 passed in 0.08s ===============================
Running distributed/pipeline/sync/test_worker ... [2021-09-20 20:00:09.384652]
Executing ['/project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_worker.py', '-v'] ... [2021-09-20 20:00:09.384775]
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.13.0 -- /project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch/test/.hypothesis/examples')
torch: 1.8.1
rootdir: /tmp/boegelbot/PyTorch/1.8.1/fosscuda-2019b-Python-3.7.4/pytorch
plugins: hypothesis-4.44.2
collecting ... collected 8 items

distributed/pipeline/sync/test_worker.py::test_join_running_workers PASSED [ 12%]
distributed/pipeline/sync/test_worker.py::test_join_running_workers_with_exception PASSED [ 25%]
distributed/pipeline/sync/test_worker.py::test_compute_multithreading PASSED [ 37%]
distributed/pipeline/sync/test_worker.py::test_compute_success PASSED    [ 50%]
distributed/pipeline/sync/test_worker.py::test_compute_exception PASSED  [ 62%]
distributed/pipeline/sync/test_worker.py::test_grad_mode[True] PASSED    [ 75%]
distributed/pipeline/sync/test_worker.py::test_grad_mode[False] PASSED   [ 87%]
distributed/pipeline/sync/test_worker.py::test_worker_per_device PASSED  [100%]

============================== 8 passed in 0.27s ===============================
Running distributed/optim/test_zero_redundancy_optimizer ... [2021-09-20 20:00:10.799535]
Executing ['/project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python', 'distributed/optim/test_zero_redundancy_optimizer.py', '-v'] ... [2021-09-20 20:00:10.799652]
test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed)
Check that ZeroRedundancyOptimizer properly handles adding a new param_group a posteriori, ... ok
test_collect_shards (__main__.TestZeroRedundancyOptimizerDistributed) ... ok
test_multiple_groups (__main__.TestZeroRedundancyOptimizerDistributed)
Check that the ZeroRedundancyOptimizer handles working with multiple process groups ... ok
test_pytorch_parity (__main__.TestZeroRedundancyOptimizerDistributed)
When combined with DDP, check that ZeroRedundancyOptimizer(optimizer) and the same monolithic optimizer ... skipped 'CUDA is not available.'
test_sharding (__main__.TestZeroRedundancyOptimizerDistributed)
Check the sharding at construction time ... ok
test_step (__main__.TestZeroRedundancyOptimizerDistributed)
Check that the ZeroRedundancyOptimizer wrapper properly exposes the `.step()` interface ... ok
test_step_with_closure (__main__.TestZeroRedundancyOptimizerDistributed)
Check that the ZeroRedundancyOptimizer wrapper properly exposes the `.step(closure)` interface ... ok
test_implicit_local_state_dict (__main__.TestZeroRedundancyOptimizerSingleRank)
Check that it's possible to pull a local state dict ... WARNING:root:Optimizer state has not been consolidated. Returning the local state
WARNING:root:Please call `consolidate_state_dict()` beforehand if you meant to save the global state
ok
test_local_state_dict (__main__.TestZeroRedundancyOptimizerSingleRank)
Check that it's possible to pull a local state dict ... ok
test_lr_scheduler (__main__.TestZeroRedundancyOptimizerSingleRank)
Check that a normal torch lr_scheduler is usable with ZeroRedundancyOptimizer ... ok
test_state_dict (__main__.TestZeroRedundancyOptimizerSingleRank)
Check that the ZeroRedundancyOptimizer exposes the expected state dict interface, ... ok
test_step_with_extra_inner_key (__main__.TestZeroRedundancyOptimizerSingleRank)
Check that an optimizer adding extra keys to the param_groups ... ok
test_step_with_kwargs (__main__.TestZeroRedundancyOptimizerSingleRank)
Check that the `step(**kwargs)` interface is properly exposed ... ok
test_step_without_closure (__main__.TestZeroRedundancyOptimizerSingleRank)
Check that the step() method (without closure) is handlded as expected ... ok
test_zero_grad (__main__.TestZeroRedundancyOptimizerSingleRank)
Check that the zero_grad attribute is properly handled ... ok

----------------------------------------------------------------------
Ran 15 tests in 16.428s

OK (skipped=1)
test_nn failed! Received signal: SIGKILL
 (at easybuild/easybuild-framework/easybuild/tools/run.py:577 in parse_cmd_output)
== 2021-09-20 20:00:28,896 build_log.py:265 INFO ... (took 1 hour 18 mins 32 secs)
== 2021-09-20 20:00:28,897 config.py:635 DEBUG software install path as specified by 'installpath' and 'subdir_software': /project/boegelbot/Rocky8/haswell/software
== 2021-09-20 20:00:28,897 filetools.py:1884 INFO Removing lock /project/boegelbot/Rocky8/haswell/software/.locks/_project_boegelbot_Rocky8_haswell_software_PyTorch_1.8.1-fosscuda-2019b-Python-3.7.4.lock...
== 2021-09-20 20:00:28,900 filetools.py:359 INFO Path /project/boegelbot/Rocky8/haswell/software/.locks/_project_boegelbot_Rocky8_haswell_software_PyTorch_1.8.1-fosscuda-2019b-Python-3.7.4.lock successfully removed.
== 2021-09-20 20:00:28,900 filetools.py:1888 INFO Lock removed: /project/boegelbot/Rocky8/haswell/software/.locks/_project_boegelbot_Rocky8_haswell_software_PyTorch_1.8.1-fosscuda-2019b-Python-3.7.4.lock
== 2021-09-20 20:00:28,900 easyblock.py:3753 WARNING build failed (first 300 chars): cmd "export PYTHONPATH=/tmp/eb-2ojgtdt7/tmpu7x25lqj/lib/python3.7/site-packages:$PYTHONPATH &&  cd test && PYTHONUNBUFFERED=1 /project/boegelbot/Rocky8/haswell/software/Python/3.7.4-GCCcore-8.3.0/bin/python run_test.py --continue-through-error  --verbose -x distributed/rpc/test_process_group_agent t
== 2021-09-20 20:00:28,901 easyblock.py:304 INFO Closing log for application name PyTorch version 1.8.1