Created
August 6, 2020 20:14
-
-
Save colesbury/7976640c82af042fd34eeb5e21322652 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| INFO:root:World size is : 8 | |
| INFO:root:Running without GIL | |
| INFO:root:Set start method of multiprocessing to spawn | |
| INFO:root:Using the config | |
| {"seed": 0, "data": {"dataset": "iamdb", "data_path": "/datasets01/iamdb/060820/", "img_height": 64}, "criterion_type": "transducer", "criterion": {"blank": true, "allow_repeats": false, "ngram": 0}, "model_type": "tds2d", "model": {"depth": 4, "tds_groups": [{"channels": 4, "num_blocks": 3, "stride": [2, 2]}, {"channels": 16, "num_blocks": 3, "stride": [2, 2]}, {"channels": 32, "num_blocks": 3, "stride": [2, 1]}, {"channels": 64, "num_blocks": 3, "stride": [2, 1]}], "kernel_size": [5, 7], "dropout": 0.1}, "optim": {"batch_size": 32, "epochs": 400, "learning_rate": 0.1, "crit_learning_rate": 0.1, "step_size": 100, "max_grad_norm": 5}} | |
| INFO:root:Loading dataset ... | |
| INFO:root:Loading model ... | |
| INFO:root:Training tds2d model with 2,644,859 parameters. | |
| INFO:root:Starting training ... | |
| INFO:root:Epoch 1 started. | |
| INFO:root:Epoch 1 complete. nUpdates 192, Loss 4.470, CER 0.931, Time 47.532 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 3.305, CER 0.817, Best Loss 3.305, Best CER 0.817 | |
| INFO:root:Timing Info: ds_fetch : 38.27ms, model_fwd : 40.80ms, crit_fwd : 28.69ms, bwd : 129.65ms, optim : 2.83ms, metrics : 7.03ms, train_total : 47532.01ms, test_total : 3674.34ms | |
| INFO:root:Epoch 2 started. | |
| INFO:root:Epoch 2 complete. nUpdates 384, Loss 3.234, CER 0.845, Time 46.452 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 3.677, CER 0.926, Best Loss 3.305, Best CER 0.817 | |
| INFO:root:Timing Info: ds_fetch : 36.60ms, model_fwd : 37.67ms, crit_fwd : 29.17ms, bwd : 128.92ms, optim : 2.18ms, metrics : 7.11ms, train_total : 46451.94ms, test_total : 3593.01ms | |
| INFO:root:Epoch 3 started. | |
| INFO:root:Epoch 3 complete. nUpdates 576, Loss 2.842, CER 0.777, Time 46.395 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 2.408, CER 0.690, Best Loss 2.408, Best CER 0.690 | |
| INFO:root:Timing Info: ds_fetch : 38.73ms, model_fwd : 37.46ms, crit_fwd : 28.41ms, bwd : 127.46ms, optim : 2.18ms, metrics : 7.11ms, train_total : 46395.50ms, test_total : 3605.94ms | |
| INFO:root:Epoch 4 started. | |
| INFO:root:Epoch 4 complete. nUpdates 768, Loss 2.528, CER 0.723, Time 46.785 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 2.285, CER 0.728, Best Loss 2.285, Best CER 0.690 | |
| INFO:root:Timing Info: ds_fetch : 38.58ms, model_fwd : 37.61ms, crit_fwd : 28.34ms, bwd : 129.35ms, optim : 2.18ms, metrics : 7.31ms, train_total : 46784.92ms, test_total : 3661.85ms | |
| INFO:root:Epoch 5 started. | |
| INFO:root:Epoch 5 complete. nUpdates 960, Loss 2.204, CER 0.646, Time 46.128 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 1.657, CER 0.484, Best Loss 1.657, Best CER 0.484 | |
| INFO:root:Timing Info: ds_fetch : 38.19ms, model_fwd : 37.40ms, crit_fwd : 28.03ms, bwd : 126.87ms, optim : 2.10ms, metrics : 7.35ms, train_total : 46127.70ms, test_total : 3684.60ms | |
| INFO:root:Epoch 6 started. | |
| INFO:root:Epoch 6 complete. nUpdates 1152, Loss 1.961, CER 0.566, Time 46.694 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 1.365, CER 0.419, Best Loss 1.365, Best CER 0.419 | |
| INFO:root:Timing Info: ds_fetch : 38.16ms, model_fwd : 37.75ms, crit_fwd : 29.33ms, bwd : 128.13ms, optim : 2.10ms, metrics : 7.44ms, train_total : 46694.23ms, test_total : 3687.88ms | |
| INFO:root:Epoch 7 started. | |
| INFO:root:Epoch 7 complete. nUpdates 1344, Loss 1.694, CER 0.492, Time 46.903 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 1.174, CER 0.337, Best Loss 1.174, Best CER 0.337 | |
| INFO:root:Timing Info: ds_fetch : 36.00ms, model_fwd : 38.27ms, crit_fwd : 29.77ms, bwd : 130.32ms, optim : 2.06ms, metrics : 7.57ms, train_total : 46902.69ms, test_total : 3694.65ms | |
| INFO:root:Epoch 8 started. | |
| INFO:root:Epoch 8 complete. nUpdates 1536, Loss 1.509, CER 0.434, Time 47.187 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 1.113, CER 0.342, Best Loss 1.113, Best CER 0.337 | |
| INFO:root:Timing Info: ds_fetch : 37.48ms, model_fwd : 38.01ms, crit_fwd : 29.63ms, bwd : 130.72ms, optim : 2.08ms, metrics : 7.55ms, train_total : 47187.17ms, test_total : 3727.71ms | |
| INFO:root:Epoch 9 started. | |
| INFO:root:Epoch 9 complete. nUpdates 1728, Loss 1.394, CER 0.399, Time 45.951 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.889, CER 0.260, Best Loss 0.889, Best CER 0.260 | |
| INFO:root:Timing Info: ds_fetch : 36.12ms, model_fwd : 36.75ms, crit_fwd : 28.28ms, bwd : 128.22ms, optim : 2.08ms, metrics : 7.60ms, train_total : 45950.68ms, test_total : 3739.38ms | |
| INFO:root:Epoch 10 started. | |
| INFO:root:Epoch 10 complete. nUpdates 1920, Loss 1.274, CER 0.366, Time 46.207 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.821, CER 0.226, Best Loss 0.821, Best CER 0.226 | |
| INFO:root:Timing Info: ds_fetch : 38.00ms, model_fwd : 37.60ms, crit_fwd : 29.95ms, bwd : 124.97ms, optim : 2.18ms, metrics : 7.65ms, train_total : 46207.30ms, test_total : 3801.52ms | |
| INFO:root:Epoch 11 started. | |
| INFO:root:Epoch 11 complete. nUpdates 2112, Loss 1.166, CER 0.335, Time 47.115 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.724, CER 0.211, Best Loss 0.724, Best CER 0.211 | |
| INFO:root:Timing Info: ds_fetch : 36.81ms, model_fwd : 37.81ms, crit_fwd : 29.82ms, bwd : 130.78ms, optim : 2.09ms, metrics : 7.80ms, train_total : 47114.80ms, test_total : 3688.74ms | |
| INFO:root:Epoch 12 started. | |
| INFO:root:Epoch 12 complete. nUpdates 2304, Loss 1.078, CER 0.312, Time 47.372 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.669, CER 0.185, Best Loss 0.669, Best CER 0.185 | |
| INFO:root:Timing Info: ds_fetch : 40.33ms, model_fwd : 37.88ms, crit_fwd : 29.01ms, bwd : 129.33ms, optim : 2.13ms, metrics : 7.74ms, train_total : 47372.05ms, test_total : 3736.71ms | |
| INFO:root:Epoch 13 started. | |
| INFO:root:Epoch 13 complete. nUpdates 2496, Loss 1.007, CER 0.291, Time 46.739 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.654, CER 0.180, Best Loss 0.654, Best CER 0.180 | |
| INFO:root:Timing Info: ds_fetch : 37.88ms, model_fwd : 37.58ms, crit_fwd : 28.75ms, bwd : 129.20ms, optim : 2.16ms, metrics : 7.57ms, train_total : 46739.39ms, test_total : 3756.95ms | |
| INFO:root:Epoch 14 started. | |
| INFO:root:Epoch 14 complete. nUpdates 2688, Loss 0.954, CER 0.275, Time 47.836 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.598, CER 0.168, Best Loss 0.598, Best CER 0.168 | |
| INFO:root:Timing Info: ds_fetch : 38.95ms, model_fwd : 37.85ms, crit_fwd : 29.20ms, bwd : 132.90ms, optim : 2.12ms, metrics : 7.83ms, train_total : 47836.05ms, test_total : 3800.32ms | |
| INFO:root:Epoch 15 started. | |
| INFO:root:Epoch 15 complete. nUpdates 2880, Loss 0.905, CER 0.261, Time 47.234 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.601, CER 0.173, Best Loss 0.598, Best CER 0.168 | |
| INFO:root:Timing Info: ds_fetch : 37.74ms, model_fwd : 37.31ms, crit_fwd : 28.82ms, bwd : 131.90ms, optim : 2.12ms, metrics : 7.83ms, train_total : 47234.32ms, test_total : 3751.95ms | |
| INFO:root:Epoch 16 started. | |
| INFO:root:Epoch 16 complete. nUpdates 3072, Loss 0.866, CER 0.249, Time 47.105 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.550, CER 0.151, Best Loss 0.550, Best CER 0.151 | |
| INFO:root:Timing Info: ds_fetch : 37.64ms, model_fwd : 37.41ms, crit_fwd : 28.88ms, bwd : 131.21ms, optim : 2.09ms, metrics : 7.83ms, train_total : 47105.51ms, test_total : 3838.30ms | |
| INFO:root:Epoch 17 started. | |
| INFO:root:Epoch 17 complete. nUpdates 3264, Loss 0.843, CER 0.244, Time 47.156 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.530, CER 0.149, Best Loss 0.530, Best CER 0.149 | |
| INFO:root:Timing Info: ds_fetch : 37.47ms, model_fwd : 37.54ms, crit_fwd : 29.32ms, bwd : 131.03ms, optim : 2.11ms, metrics : 7.85ms, train_total : 47156.50ms, test_total : 3807.74ms | |
| INFO:root:Epoch 18 started. | |
| INFO:root:Epoch 18 complete. nUpdates 3456, Loss 0.835, CER 0.239, Time 48.185 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.503, CER 0.138, Best Loss 0.503, Best CER 0.138 | |
| INFO:root:Timing Info: ds_fetch : 38.06ms, model_fwd : 37.89ms, crit_fwd : 29.70ms, bwd : 135.00ms, optim : 2.09ms, metrics : 7.95ms, train_total : 48185.15ms, test_total : 3895.67ms | |
| INFO:root:Epoch 19 started. | |
| INFO:root:Epoch 19 complete. nUpdates 3648, Loss 0.770, CER 0.224, Time 47.848 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.501, CER 0.139, Best Loss 0.501, Best CER 0.138 | |
| INFO:root:Timing Info: ds_fetch : 38.90ms, model_fwd : 37.76ms, crit_fwd : 29.74ms, bwd : 132.57ms, optim : 2.05ms, metrics : 7.90ms, train_total : 47848.57ms, test_total : 3789.76ms | |
| INFO:root:Epoch 20 started. | |
| INFO:root:Epoch 20 complete. nUpdates 3840, Loss 0.740, CER 0.215, Time 47.471 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.484, CER 0.137, Best Loss 0.484, Best CER 0.137 | |
| INFO:root:Timing Info: ds_fetch : 37.24ms, model_fwd : 37.88ms, crit_fwd : 30.16ms, bwd : 131.64ms, optim : 2.13ms, metrics : 7.88ms, train_total : 47471.44ms, test_total : 3813.85ms | |
| INFO:root:Epoch 21 started. | |
| INFO:root:Epoch 21 complete. nUpdates 4032, Loss 0.734, CER 0.212, Time 47.454 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.455, CER 0.125, Best Loss 0.455, Best CER 0.125 | |
| INFO:root:Timing Info: ds_fetch : 38.51ms, model_fwd : 37.84ms, crit_fwd : 31.24ms, bwd : 129.13ms, optim : 2.07ms, metrics : 8.07ms, train_total : 47453.77ms, test_total : 3727.41ms | |
| INFO:root:Epoch 22 started. | |
| INFO:root:Epoch 22 complete. nUpdates 4224, Loss 0.708, CER 0.206, Time 47.658 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.474, CER 0.131, Best Loss 0.455, Best CER 0.125 | |
| INFO:root:Timing Info: ds_fetch : 36.97ms, model_fwd : 37.61ms, crit_fwd : 29.65ms, bwd : 133.70ms, optim : 2.08ms, metrics : 7.93ms, train_total : 47658.36ms, test_total : 3768.50ms | |
| INFO:root:Epoch 23 started. | |
| INFO:root:Epoch 23 complete. nUpdates 4416, Loss 0.696, CER 0.202, Time 46.635 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.436, CER 0.121, Best Loss 0.436, Best CER 0.121 | |
| INFO:root:Timing Info: ds_fetch : 36.78ms, model_fwd : 37.37ms, crit_fwd : 28.48ms, bwd : 129.92ms, optim : 2.08ms, metrics : 7.97ms, train_total : 46635.25ms, test_total : 3926.95ms | |
| INFO:root:Epoch 24 started. | |
| INFO:root:Epoch 24 complete. nUpdates 4608, Loss 0.668, CER 0.193, Time 46.862 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.432, CER 0.117, Best Loss 0.432, Best CER 0.117 | |
| INFO:root:Timing Info: ds_fetch : 37.79ms, model_fwd : 37.75ms, crit_fwd : 29.62ms, bwd : 128.54ms, optim : 2.15ms, metrics : 7.94ms, train_total : 46862.00ms, test_total : 3945.65ms | |
| INFO:root:Epoch 25 started. | |
| INFO:root:Epoch 25 complete. nUpdates 4800, Loss 0.655, CER 0.190, Time 47.151 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.449, CER 0.124, Best Loss 0.432, Best CER 0.117 | |
| INFO:root:Timing Info: ds_fetch : 37.86ms, model_fwd : 37.21ms, crit_fwd : 29.12ms, bwd : 131.05ms, optim : 2.08ms, metrics : 7.96ms, train_total : 47151.55ms, test_total : 3813.09ms | |
| INFO:root:Epoch 26 started. | |
| INFO:root:Epoch 26 complete. nUpdates 4992, Loss 0.638, CER 0.185, Time 47.770 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.424, CER 0.116, Best Loss 0.424, Best CER 0.116 | |
| INFO:root:Timing Info: ds_fetch : 37.66ms, model_fwd : 37.78ms, crit_fwd : 28.94ms, bwd : 134.21ms, optim : 2.03ms, metrics : 7.89ms, train_total : 47769.78ms, test_total : 3981.84ms | |
| INFO:root:Epoch 27 started. | |
| INFO:root:Epoch 27 complete. nUpdates 5184, Loss 0.614, CER 0.179, Time 47.183 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.414, CER 0.113, Best Loss 0.414, Best CER 0.113 | |
| INFO:root:Timing Info: ds_fetch : 39.56ms, model_fwd : 37.35ms, crit_fwd : 29.56ms, bwd : 128.98ms, optim : 2.15ms, metrics : 7.84ms, train_total : 47182.71ms, test_total : 3850.78ms | |
| INFO:root:Epoch 28 started. | |
| INFO:root:Epoch 28 complete. nUpdates 5376, Loss 0.620, CER 0.181, Time 47.921 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.431, CER 0.114, Best Loss 0.414, Best CER 0.113 | |
| INFO:root:Timing Info: ds_fetch : 38.32ms, model_fwd : 38.26ms, crit_fwd : 30.96ms, bwd : 131.41ms, optim : 2.15ms, metrics : 8.19ms, train_total : 47921.20ms, test_total : 3751.84ms | |
| INFO:root:Epoch 29 started. | |
| INFO:root:Epoch 29 complete. nUpdates 5568, Loss 0.598, CER 0.174, Time 47.869 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.406, CER 0.110, Best Loss 0.406, Best CER 0.110 | |
| INFO:root:Timing Info: ds_fetch : 37.72ms, model_fwd : 38.06ms, crit_fwd : 30.93ms, bwd : 132.30ms, optim : 2.07ms, metrics : 7.93ms, train_total : 47869.22ms, test_total : 3763.76ms | |
| INFO:root:Epoch 30 started. | |
| INFO:root:Epoch 30 complete. nUpdates 5760, Loss 0.600, CER 0.175, Time 46.575 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.397, CER 0.106, Best Loss 0.397, Best CER 0.106 | |
| INFO:root:Timing Info: ds_fetch : 36.84ms, model_fwd : 37.07ms, crit_fwd : 29.23ms, bwd : 129.23ms, optim : 2.03ms, metrics : 7.89ms, train_total : 46574.90ms, test_total : 3851.98ms | |
| INFO:root:Epoch 31 started. | |
| INFO:root:Epoch 31 complete. nUpdates 5952, Loss 0.601, CER 0.174, Time 46.679 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.394, CER 0.106, Best Loss 0.394, Best CER 0.106 | |
| INFO:root:Timing Info: ds_fetch : 36.58ms, model_fwd : 36.92ms, crit_fwd : 29.56ms, bwd : 129.77ms, optim : 2.04ms, metrics : 7.97ms, train_total : 46679.52ms, test_total : 3743.29ms | |
| INFO:root:Epoch 32 started. | |
| INFO:root:Epoch 32 complete. nUpdates 6144, Loss 0.569, CER 0.167, Time 47.416 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.388, CER 0.105, Best Loss 0.388, Best CER 0.105 | |
| INFO:root:Timing Info: ds_fetch : 39.58ms, model_fwd : 37.23ms, crit_fwd : 30.18ms, bwd : 129.45ms, optim : 2.09ms, metrics : 8.13ms, train_total : 47416.08ms, test_total : 3906.03ms | |
| INFO:root:Epoch 33 started. | |
| INFO:root:Epoch 33 complete. nUpdates 6336, Loss 0.577, CER 0.168, Time 48.301 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.393, CER 0.107, Best Loss 0.388, Best CER 0.105 | |
| INFO:root:Timing Info: ds_fetch : 40.68ms, model_fwd : 38.06ms, crit_fwd : 31.43ms, bwd : 130.72ms, optim : 2.17ms, metrics : 8.20ms, train_total : 48301.39ms, test_total : 3779.26ms | |
| INFO:root:Epoch 34 started. | |
| INFO:root:Epoch 34 complete. nUpdates 6528, Loss 0.558, CER 0.164, Time 47.779 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.380, CER 0.101, Best Loss 0.380, Best CER 0.101 | |
| INFO:root:Timing Info: ds_fetch : 39.90ms, model_fwd : 37.80ms, crit_fwd : 30.28ms, bwd : 130.35ms, optim : 2.20ms, metrics : 8.02ms, train_total : 47779.37ms, test_total : 4032.06ms | |
| INFO:root:Epoch 35 started. | |
| INFO:root:Epoch 35 complete. nUpdates 6720, Loss 0.549, CER 0.160, Time 48.209 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.381, CER 0.102, Best Loss 0.380, Best CER 0.101 | |
| INFO:root:Timing Info: ds_fetch : 38.46ms, model_fwd : 37.91ms, crit_fwd : 29.47ms, bwd : 134.92ms, optim : 2.06ms, metrics : 7.98ms, train_total : 48209.58ms, test_total : 3874.04ms | |
| INFO:root:Epoch 36 started. | |
| INFO:root:Epoch 36 complete. nUpdates 6912, Loss 0.531, CER 0.155, Time 47.826 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.378, CER 0.101, Best Loss 0.378, Best CER 0.101 | |
| INFO:root:Timing Info: ds_fetch : 39.04ms, model_fwd : 37.97ms, crit_fwd : 30.85ms, bwd : 130.67ms, optim : 2.15ms, metrics : 8.11ms, train_total : 47826.28ms, test_total : 4049.82ms | |
| INFO:root:Epoch 37 started. | |
| INFO:root:Epoch 37 complete. nUpdates 7104, Loss 0.537, CER 0.156, Time 47.824 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.363, CER 0.098, Best Loss 0.363, Best CER 0.098 | |
| INFO:root:Timing Info: ds_fetch : 36.07ms, model_fwd : 37.85ms, crit_fwd : 31.46ms, bwd : 133.13ms, optim : 2.05ms, metrics : 8.25ms, train_total : 47824.25ms, test_total : 3807.32ms | |
| INFO:root:Epoch 38 started. | |
| INFO:root:Epoch 38 complete. nUpdates 7296, Loss 0.511, CER 0.150, Time 47.042 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.353, CER 0.096, Best Loss 0.353, Best CER 0.096 | |
| INFO:root:Timing Info: ds_fetch : 36.44ms, model_fwd : 37.84ms, crit_fwd : 30.46ms, bwd : 129.57ms, optim : 2.17ms, metrics : 8.24ms, train_total : 47041.84ms, test_total : 3933.40ms | |
| INFO:root:Epoch 39 started. | |
| INFO:root:Epoch 39 complete. nUpdates 7488, Loss 0.519, CER 0.152, Time 48.108 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.356, CER 0.095, Best Loss 0.353, Best CER 0.095 | |
| INFO:root:Timing Info: ds_fetch : 39.33ms, model_fwd : 37.89ms, crit_fwd : 29.88ms, bwd : 132.69ms, optim : 2.09ms, metrics : 8.38ms, train_total : 48108.26ms, test_total : 3964.27ms | |
| INFO:root:Epoch 40 started. | |
| INFO:root:Epoch 40 complete. nUpdates 7680, Loss 0.515, CER 0.151, Time 48.245 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.360, CER 0.097, Best Loss 0.353, Best CER 0.095 | |
| INFO:root:Timing Info: ds_fetch : 38.00ms, model_fwd : 38.14ms, crit_fwd : 30.62ms, bwd : 133.90ms, optim : 2.09ms, metrics : 8.25ms, train_total : 48244.73ms, test_total : 3914.11ms | |
| INFO:root:Epoch 41 started. | |
| INFO:root:Epoch 41 complete. nUpdates 7872, Loss 0.507, CER 0.148, Time 47.743 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.350, CER 0.094, Best Loss 0.350, Best CER 0.094 | |
| INFO:root:Timing Info: ds_fetch : 40.88ms, model_fwd : 37.31ms, crit_fwd : 30.08ms, bwd : 130.03ms, optim : 2.04ms, metrics : 8.03ms, train_total : 47743.03ms, test_total : 3965.88ms | |
| INFO:root:Epoch 42 started. | |
| INFO:root:Epoch 42 complete. nUpdates 8064, Loss 0.492, CER 0.145, Time 47.866 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.352, CER 0.095, Best Loss 0.350, Best CER 0.094 | |
| INFO:root:Timing Info: ds_fetch : 36.84ms, model_fwd : 38.04ms, crit_fwd : 30.79ms, bwd : 133.19ms, optim : 2.09ms, metrics : 8.05ms, train_total : 47866.50ms, test_total : 3817.52ms | |
| INFO:root:Epoch 43 started. | |
| INFO:root:Epoch 43 complete. nUpdates 8256, Loss 0.501, CER 0.146, Time 47.405 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.346, CER 0.093, Best Loss 0.346, Best CER 0.093 | |
| INFO:root:Timing Info: ds_fetch : 37.64ms, model_fwd : 38.48ms, crit_fwd : 30.74ms, bwd : 129.44ms, optim : 2.14ms, metrics : 8.18ms, train_total : 47404.87ms, test_total : 4004.43ms | |
| INFO:root:Epoch 44 started. | |
| INFO:root:Epoch 44 complete. nUpdates 8448, Loss 0.503, CER 0.147, Time 47.743 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.358, CER 0.096, Best Loss 0.346, Best CER 0.093 | |
| INFO:root:Timing Info: ds_fetch : 39.31ms, model_fwd : 37.52ms, crit_fwd : 29.92ms, bwd : 131.42ms, optim : 2.11ms, metrics : 8.09ms, train_total : 47742.64ms, test_total : 3941.07ms | |
| INFO:root:Epoch 45 started. | |
| INFO:root:Epoch 45 complete. nUpdates 8640, Loss 0.471, CER 0.138, Time 47.982 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.342, CER 0.092, Best Loss 0.342, Best CER 0.092 | |
| INFO:root:Timing Info: ds_fetch : 40.80ms, model_fwd : 37.72ms, crit_fwd : 30.03ms, bwd : 130.74ms, optim : 2.06ms, metrics : 8.25ms, train_total : 47981.91ms, test_total : 3861.01ms | |
| INFO:root:Epoch 46 started. | |
| INFO:root:Epoch 46 complete. nUpdates 8832, Loss 0.494, CER 0.144, Time 47.786 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.337, CER 0.091, Best Loss 0.337, Best CER 0.091 | |
| INFO:root:Timing Info: ds_fetch : 39.22ms, model_fwd : 38.03ms, crit_fwd : 32.29ms, bwd : 128.46ms, optim : 2.08ms, metrics : 8.50ms, train_total : 47786.61ms, test_total : 3895.54ms | |
| INFO:root:Epoch 47 started. | |
| INFO:root:Epoch 47 complete. nUpdates 9024, Loss 0.489, CER 0.144, Time 47.350 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.339, CER 0.090, Best Loss 0.337, Best CER 0.090 | |
| INFO:root:Timing Info: ds_fetch : 38.02ms, model_fwd : 37.68ms, crit_fwd : 29.73ms, bwd : 130.70ms, optim : 2.12ms, metrics : 8.07ms, train_total : 47349.66ms, test_total : 3978.92ms | |
| INFO:root:Epoch 48 started. | |
| INFO:root:Epoch 48 complete. nUpdates 9216, Loss 0.467, CER 0.137, Time 48.605 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.346, CER 0.095, Best Loss 0.337, Best CER 0.090 | |
| INFO:root:Timing Info: ds_fetch : 38.16ms, model_fwd : 38.56ms, crit_fwd : 33.00ms, bwd : 132.30ms, optim : 2.19ms, metrics : 8.64ms, train_total : 48604.90ms, test_total : 3920.85ms | |
| INFO:root:Epoch 49 started. | |
| INFO:root:Epoch 49 complete. nUpdates 9408, Loss 0.478, CER 0.141, Time 48.011 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.337, CER 0.091, Best Loss 0.337, Best CER 0.090 | |
| INFO:root:Timing Info: ds_fetch : 38.52ms, model_fwd : 38.13ms, crit_fwd : 31.02ms, bwd : 131.64ms, optim : 2.13ms, metrics : 8.32ms, train_total : 48011.18ms, test_total : 3873.95ms | |
| INFO:root:Epoch 50 started. | |
| INFO:root:Epoch 50 complete. nUpdates 9600, Loss 0.472, CER 0.139, Time 47.491 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.345, CER 0.093, Best Loss 0.337, Best CER 0.090 | |
| INFO:root:Timing Info: ds_fetch : 36.77ms, model_fwd : 36.89ms, crit_fwd : 30.75ms, bwd : 132.48ms, optim : 2.06ms, metrics : 8.11ms, train_total : 47490.77ms, test_total : 3825.25ms | |
| INFO:root:Epoch 51 started. | |
| INFO:root:Epoch 51 complete. nUpdates 9792, Loss 0.459, CER 0.134, Time 48.187 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.334, CER 0.089, Best Loss 0.334, Best CER 0.089 | |
| INFO:root:Timing Info: ds_fetch : 39.44ms, model_fwd : 37.60ms, crit_fwd : 30.26ms, bwd : 133.13ms, optim : 2.06ms, metrics : 8.17ms, train_total : 48186.70ms, test_total : 3958.90ms | |
| INFO:root:Epoch 52 started. | |
| INFO:root:Epoch 52 complete. nUpdates 9984, Loss 0.451, CER 0.132, Time 47.840 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.336, CER 0.091, Best Loss 0.334, Best CER 0.089 | |
| INFO:root:Timing Info: ds_fetch : 38.41ms, model_fwd : 38.17ms, crit_fwd : 31.53ms, bwd : 130.36ms, optim : 2.20ms, metrics : 8.19ms, train_total : 47839.91ms, test_total : 4020.39ms | |
| INFO:root:Epoch 53 started. | |
| INFO:root:Epoch 53 complete. nUpdates 10176, Loss 0.451, CER 0.133, Time 47.741 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.333, CER 0.089, Best Loss 0.333, Best CER 0.089 | |
| INFO:root:Timing Info: ds_fetch : 37.92ms, model_fwd : 38.43ms, crit_fwd : 32.59ms, bwd : 128.90ms, optim : 2.08ms, metrics : 8.43ms, train_total : 47741.46ms, test_total : 3950.42ms | |
| INFO:root:Epoch 54 started. | |
| INFO:root:Epoch 54 complete. nUpdates 10368, Loss 0.433, CER 0.128, Time 48.118 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.332, CER 0.089, Best Loss 0.332, Best CER 0.089 | |
| INFO:root:Timing Info: ds_fetch : 37.11ms, model_fwd : 37.93ms, crit_fwd : 31.60ms, bwd : 133.19ms, optim : 2.05ms, metrics : 8.45ms, train_total : 48118.32ms, test_total : 3844.73ms | |
| INFO:root:Epoch 55 started. | |
| INFO:root:Epoch 55 complete. nUpdates 10560, Loss 0.456, CER 0.134, Time 47.926 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.331, CER 0.090, Best Loss 0.331, Best CER 0.089 | |
| INFO:root:Timing Info: ds_fetch : 39.02ms, model_fwd : 37.68ms, crit_fwd : 29.99ms, bwd : 132.39ms, optim : 2.09ms, metrics : 8.15ms, train_total : 47925.80ms, test_total : 3933.52ms | |
| INFO:root:Epoch 56 started. | |
| INFO:root:Epoch 56 complete. nUpdates 10752, Loss 0.439, CER 0.130, Time 47.311 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.327, CER 0.087, Best Loss 0.327, Best CER 0.087 | |
| INFO:root:Timing Info: ds_fetch : 37.68ms, model_fwd : 37.39ms, crit_fwd : 31.59ms, bwd : 128.98ms, optim : 2.11ms, metrics : 8.36ms, train_total : 47311.08ms, test_total : 4023.32ms | |
| INFO:root:Epoch 57 started. | |
| INFO:root:Epoch 57 complete. nUpdates 10944, Loss 0.429, CER 0.127, Time 48.314 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.323, CER 0.088, Best Loss 0.323, Best CER 0.087 | |
| INFO:root:Timing Info: ds_fetch : 37.52ms, model_fwd : 37.75ms, crit_fwd : 30.26ms, bwd : 135.51ms, optim : 2.08ms, metrics : 8.24ms, train_total : 48314.39ms, test_total : 4099.41ms | |
| INFO:root:Epoch 58 started. | |
| INFO:root:Epoch 58 complete. nUpdates 11136, Loss 0.454, CER 0.133, Time 48.573 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.325, CER 0.088, Best Loss 0.323, Best CER 0.087 | |
| INFO:root:Timing Info: ds_fetch : 38.18ms, model_fwd : 38.57ms, crit_fwd : 32.82ms, bwd : 132.43ms, optim : 2.09ms, metrics : 8.60ms, train_total : 48572.68ms, test_total : 3962.61ms | |
| INFO:root:Epoch 59 started. | |
| INFO:root:Epoch 59 complete. nUpdates 11328, Loss 0.417, CER 0.123, Time 47.815 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.333, CER 0.088, Best Loss 0.323, Best CER 0.087 | |
| INFO:root:Timing Info: ds_fetch : 37.28ms, model_fwd : 37.60ms, crit_fwd : 31.00ms, bwd : 132.45ms, optim : 2.13ms, metrics : 8.28ms, train_total : 47814.84ms, test_total : 4077.95ms | |
| INFO:root:Epoch 60 started. | |
| INFO:root:Epoch 60 complete. nUpdates 11520, Loss 0.424, CER 0.125, Time 47.770 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.320, CER 0.085, Best Loss 0.320, Best CER 0.085 | |
| INFO:root:Timing Info: ds_fetch : 39.56ms, model_fwd : 37.59ms, crit_fwd : 29.86ms, bwd : 131.19ms, optim : 2.15ms, metrics : 8.12ms, train_total : 47769.93ms, test_total : 4006.16ms | |
| INFO:root:Epoch 61 started. | |
| INFO:root:Epoch 61 complete. nUpdates 11712, Loss 0.428, CER 0.127, Time 47.970 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.324, CER 0.086, Best Loss 0.320, Best CER 0.085 | |
| INFO:root:Timing Info: ds_fetch : 41.22ms, model_fwd : 37.45ms, crit_fwd : 30.12ms, bwd : 130.55ms, optim : 2.04ms, metrics : 8.16ms, train_total : 47970.11ms, test_total : 3896.87ms | |
| INFO:root:Epoch 62 started. | |
| INFO:root:Epoch 62 complete. nUpdates 11904, Loss 0.428, CER 0.127, Time 48.509 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.323, CER 0.087, Best Loss 0.320, Best CER 0.085 | |
| INFO:root:Timing Info: ds_fetch : 37.22ms, model_fwd : 38.02ms, crit_fwd : 31.12ms, bwd : 135.62ms, optim : 2.02ms, metrics : 8.36ms, train_total : 48508.77ms, test_total : 4138.91ms | |
| INFO:root:Epoch 63 started. | |
| INFO:root:Epoch 63 complete. nUpdates 12096, Loss 0.416, CER 0.123, Time 48.369 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.323, CER 0.086, Best Loss 0.320, Best CER 0.085 | |
| INFO:root:Timing Info: ds_fetch : 39.39ms, model_fwd : 38.00ms, crit_fwd : 30.93ms, bwd : 133.02ms, optim : 2.06ms, metrics : 8.22ms, train_total : 48369.12ms, test_total : 4021.88ms | |
| INFO:root:Epoch 64 started. | |
| INFO:root:Epoch 64 complete. nUpdates 12288, Loss 0.407, CER 0.121, Time 48.392 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.317, CER 0.085, Best Loss 0.317, Best CER 0.085 | |
| INFO:root:Timing Info: ds_fetch : 41.86ms, model_fwd : 37.79ms, crit_fwd : 30.95ms, bwd : 130.95ms, optim : 2.07ms, metrics : 8.10ms, train_total : 48392.35ms, test_total : 3932.58ms | |
| INFO:root:Epoch 65 started. | |
| INFO:root:Epoch 65 complete. nUpdates 12480, Loss 0.407, CER 0.121, Time 47.736 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.324, CER 0.086, Best Loss 0.317, Best CER 0.085 | |
| INFO:root:Timing Info: ds_fetch : 38.86ms, model_fwd : 37.83ms, crit_fwd : 32.11ms, bwd : 129.16ms, optim : 2.06ms, metrics : 8.31ms, train_total : 47736.03ms, test_total : 4069.11ms | |
| INFO:root:Epoch 66 started. | |
| INFO:root:Epoch 66 complete. nUpdates 12672, Loss 0.412, CER 0.121, Time 47.684 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.314, CER 0.085, Best Loss 0.314, Best CER 0.085 | |
| INFO:root:Timing Info: ds_fetch : 38.86ms, model_fwd : 37.25ms, crit_fwd : 31.21ms, bwd : 130.12ms, optim : 2.12ms, metrics : 8.50ms, train_total : 47683.70ms, test_total : 3989.37ms | |
| INFO:root:Epoch 67 started. | |
| INFO:root:Epoch 67 complete. nUpdates 12864, Loss 0.399, CER 0.118, Time 48.281 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.315, CER 0.084, Best Loss 0.314, Best CER 0.084 | |
| INFO:root:Timing Info: ds_fetch : 39.37ms, model_fwd : 38.02ms, crit_fwd : 30.35ms, bwd : 133.14ms, optim : 2.09ms, metrics : 8.20ms, train_total : 48281.59ms, test_total : 4005.26ms | |
| INFO:root:Epoch 68 started. | |
| INFO:root:Epoch 68 complete. nUpdates 13056, Loss 0.416, CER 0.122, Time 47.720 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.318, CER 0.085, Best Loss 0.314, Best CER 0.084 | |
| INFO:root:Timing Info: ds_fetch : 36.90ms, model_fwd : 38.16ms, crit_fwd : 31.76ms, bwd : 131.17ms, optim : 2.09ms, metrics : 8.16ms, train_total : 47720.18ms, test_total : 3962.35ms | |
| INFO:root:Epoch 69 started. | |
| INFO:root:Epoch 69 complete. nUpdates 13248, Loss 0.396, CER 0.118, Time 48.670 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.311, CER 0.082, Best Loss 0.311, Best CER 0.082 | |
| INFO:root:Timing Info: ds_fetch : 37.61ms, model_fwd : 38.65ms, crit_fwd : 33.01ms, bwd : 133.38ms, optim : 2.13ms, metrics : 8.41ms, train_total : 48670.07ms, test_total : 4122.49ms | |
| INFO:root:Epoch 70 started. | |
| INFO:root:Epoch 70 complete. nUpdates 13440, Loss 0.392, CER 0.116, Time 47.159 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.316, CER 0.083, Best Loss 0.311, Best CER 0.082 | |
| INFO:root:Timing Info: ds_fetch : 37.42ms, model_fwd : 36.99ms, crit_fwd : 29.68ms, bwd : 131.03ms, optim : 2.08ms, metrics : 8.14ms, train_total : 47159.60ms, test_total : 4157.84ms | |
| INFO:root:Epoch 71 started. | |
| INFO:root:Epoch 71 complete. nUpdates 13632, Loss 0.390, CER 0.116, Time 47.837 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.315, CER 0.084, Best Loss 0.311, Best CER 0.082 | |
| INFO:root:Timing Info: ds_fetch : 35.58ms, model_fwd : 37.60ms, crit_fwd : 31.62ms, bwd : 133.65ms, optim : 2.11ms, metrics : 8.32ms, train_total : 47837.57ms, test_total : 4215.69ms | |
| INFO:root:Epoch 72 started. | |
| INFO:root:Epoch 72 complete. nUpdates 13824, Loss 0.414, CER 0.122, Time 48.073 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.323, CER 0.086, Best Loss 0.311, Best CER 0.082 | |
| INFO:root:Timing Info: ds_fetch : 37.72ms, model_fwd : 37.35ms, crit_fwd : 31.09ms, bwd : 133.38ms, optim : 2.12ms, metrics : 8.43ms, train_total : 48073.05ms, test_total : 4055.53ms | |
| INFO:root:Epoch 73 started. | |
| INFO:root:Epoch 73 complete. nUpdates 14016, Loss 0.385, CER 0.115, Time 48.490 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.315, CER 0.083, Best Loss 0.311, Best CER 0.082 | |
| INFO:root:Timing Info: ds_fetch : 40.50ms, model_fwd : 37.75ms, crit_fwd : 31.53ms, bwd : 132.10ms, optim : 2.14ms, metrics : 8.22ms, train_total : 48490.06ms, test_total : 3940.73ms | |
| INFO:root:Epoch 74 started. | |
| INFO:root:Epoch 74 complete. nUpdates 14208, Loss 0.401, CER 0.119, Time 47.558 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.310, CER 0.081, Best Loss 0.310, Best CER 0.081 | |
| INFO:root:Timing Info: ds_fetch : 37.25ms, model_fwd : 37.39ms, crit_fwd : 29.95ms, bwd : 132.63ms, optim : 2.10ms, metrics : 8.09ms, train_total : 47558.28ms, test_total : 4083.81ms | |
| INFO:root:Epoch 75 started. | |
| INFO:root:Epoch 75 complete. nUpdates 14400, Loss 0.391, CER 0.116, Time 48.411 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.315, CER 0.083, Best Loss 0.310, Best CER 0.081 | |
| INFO:root:Timing Info: ds_fetch : 39.58ms, model_fwd : 37.53ms, crit_fwd : 31.32ms, bwd : 132.98ms, optim : 2.11ms, metrics : 8.31ms, train_total : 48411.38ms, test_total : 3917.95ms | |
| INFO:root:Epoch 76 started. | |
| INFO:root:Epoch 76 complete. nUpdates 14592, Loss 0.393, CER 0.117, Time 48.030 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.318, CER 0.084, Best Loss 0.310, Best CER 0.081 | |
| INFO:root:Timing Info: ds_fetch : 37.91ms, model_fwd : 37.82ms, crit_fwd : 30.67ms, bwd : 133.10ms, optim : 2.06ms, metrics : 8.30ms, train_total : 48029.77ms, test_total : 4055.03ms | |
| INFO:root:Epoch 77 started. | |
| INFO:root:Epoch 77 complete. nUpdates 14784, Loss 0.378, CER 0.113, Time 47.312 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.305, CER 0.081, Best Loss 0.305, Best CER 0.081 | |
| INFO:root:Timing Info: ds_fetch : 37.99ms, model_fwd : 37.22ms, crit_fwd : 29.87ms, bwd : 130.65ms, optim : 2.16ms, metrics : 8.23ms, train_total : 47312.50ms, test_total : 4112.51ms | |
| INFO:root:Epoch 78 started. | |
| INFO:root:Epoch 78 complete. nUpdates 14976, Loss 0.387, CER 0.115, Time 48.391 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.310, CER 0.081, Best Loss 0.305, Best CER 0.081 | |
| INFO:root:Timing Info: ds_fetch : 37.82ms, model_fwd : 38.07ms, crit_fwd : 31.54ms, bwd : 133.91ms, optim : 2.08ms, metrics : 8.33ms, train_total : 48390.71ms, test_total : 4049.45ms | |
| INFO:root:Epoch 79 started. | |
| INFO:root:Epoch 79 complete. nUpdates 15168, Loss 0.377, CER 0.112, Time 48.702 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.310, CER 0.083, Best Loss 0.305, Best CER 0.081 | |
| INFO:root:Timing Info: ds_fetch : 39.36ms, model_fwd : 38.03ms, crit_fwd : 31.51ms, bwd : 134.03ms, optim : 2.08ms, metrics : 8.35ms, train_total : 48702.22ms, test_total : 3923.78ms | |
| INFO:root:Epoch 80 started. | |
| INFO:root:Epoch 80 complete. nUpdates 15360, Loss 0.384, CER 0.115, Time 48.539 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.306, CER 0.081, Best Loss 0.305, Best CER 0.081 | |
| INFO:root:Timing Info: ds_fetch : 37.89ms, model_fwd : 38.06ms, crit_fwd : 31.41ms, bwd : 134.80ms, optim : 2.05ms, metrics : 8.31ms, train_total : 48539.51ms, test_total : 4094.73ms | |
| INFO:root:Epoch 81 started. | |
| INFO:root:Epoch 81 complete. nUpdates 15552, Loss 0.375, CER 0.111, Time 48.401 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.300, CER 0.080, Best Loss 0.300, Best CER 0.080 | |
| INFO:root:Timing Info: ds_fetch : 38.92ms, model_fwd : 37.42ms, crit_fwd : 31.14ms, bwd : 133.86ms, optim : 2.13ms, metrics : 8.33ms, train_total : 48400.69ms, test_total : 3993.04ms | |
| INFO:root:Epoch 82 started. | |
| INFO:root:Epoch 82 complete. nUpdates 15744, Loss 0.379, CER 0.112, Time 49.149 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.305, CER 0.080, Best Loss 0.300, Best CER 0.080 | |
| INFO:root:Timing Info: ds_fetch : 42.90ms, model_fwd : 37.99ms, crit_fwd : 31.47ms, bwd : 132.85ms, optim : 2.07ms, metrics : 8.38ms, train_total : 49149.14ms, test_total : 4015.95ms | |
| INFO:root:Epoch 83 started. | |
| INFO:root:Epoch 83 complete. nUpdates 15936, Loss 0.365, CER 0.109, Time 48.848 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.331, CER 0.087, Best Loss 0.300, Best CER 0.080 | |
| INFO:root:Timing Info: ds_fetch : 39.26ms, model_fwd : 37.80ms, crit_fwd : 31.94ms, bwd : 134.77ms, optim : 2.10ms, metrics : 8.25ms, train_total : 48847.65ms, test_total : 4143.01ms | |
| INFO:root:Epoch 84 started. | |
| INFO:root:Epoch 84 complete. nUpdates 16128, Loss 0.367, CER 0.109, Time 48.223 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.302, CER 0.080, Best Loss 0.300, Best CER 0.080 | |
| INFO:root:Timing Info: ds_fetch : 37.63ms, model_fwd : 37.48ms, crit_fwd : 30.78ms, bwd : 134.49ms, optim : 2.14ms, metrics : 8.35ms, train_total : 48223.02ms, test_total : 3976.46ms | |
| INFO:root:Epoch 85 started. | |
| INFO:root:Epoch 85 complete. nUpdates 16320, Loss 0.373, CER 0.111, Time 48.125 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.300, CER 0.078, Best Loss 0.300, Best CER 0.078 | |
| INFO:root:Timing Info: ds_fetch : 36.82ms, model_fwd : 37.92ms, crit_fwd : 31.14ms, bwd : 133.93ms, optim : 2.21ms, metrics : 8.34ms, train_total : 48125.02ms, test_total : 4280.53ms | |
| INFO:root:Epoch 86 started. | |
| INFO:root:Epoch 86 complete. nUpdates 16512, Loss 0.366, CER 0.109, Time 48.096 (s) | |
| INFO:root:Evaluating validation set.. | |
| INFO:root:Validation Set: Loss 0.301, CER 0.079, Best Loss 0.300, Best CER 0.078 | |
| INFO:root:Timing Info: ds_fetch : 38.60ms, model_fwd : 37.71ms, crit_fwd : 31.53ms, bwd : 131.95ms, optim : 2.10ms, metrics : 8.32ms, train_total : 48096.61ms, test_total : 4084.42ms | |
| INFO:root:Epoch 87 started. | |
| Traceback (most recent call last): | |
| File "<string>", line 1, in <module> | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main | |
| exitcode = _main(fd, parent_sentinel) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 126, in _main | |
| self = reduction.pickle.load(from_parent) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__ | |
| self._semlock = _multiprocessing.SemLock._rebuild(*state) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "<string>", line 1, in <module> | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main | |
| exitcode = _main(fd, parent_sentinel) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 126, in _main | |
| self = reduction.pickle.load(from_parent) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__ | |
| self._semlock = _multiprocessing.SemLock._rebuild(*state) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "<string>", line 1, in <module> | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main | |
| exitcode = _main(fd, parent_sentinel) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 126, in _main | |
| self = reduction.pickle.load(from_parent) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__ | |
| self._semlock = _multiprocessing.SemLock._rebuild(*state) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "<string>", line 1, in <module> | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main | |
| exitcode = _main(fd, parent_sentinel) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 126, in _main | |
| self = reduction.pickle.load(from_parent) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__ | |
| self._semlock = _multiprocessing.SemLock._rebuild(*state) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58336) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58336) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "<string>", line 1, in <module> | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main | |
| exitcode = _main(fd, parent_sentinel) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 126, in _main | |
| self = reduction.pickle.load(from_parent) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__ | |
| self._semlock = _multiprocessing.SemLock._rebuild(*state) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58337) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58337) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58339) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58339) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58340) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58340) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "<string>", line 1, in <module> | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main | |
| exitcode = _main(fd, parent_sentinel) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 126, in _main | |
| self = reduction.pickle.load(from_parent) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__ | |
| self._semlock = _multiprocessing.SemLock._rebuild(*state) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "<string>", line 1, in <module> | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main | |
| exitcode = _main(fd, parent_sentinel) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 126, in _main | |
| self = reduction.pickle.load(from_parent) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__ | |
| self._semlock = _multiprocessing.SemLock._rebuild(*state) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58342) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58342) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "<string>", line 1, in <module> | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main | |
| exitcode = _main(fd, parent_sentinel) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/spawn.py", line 126, in _main | |
| self = reduction.pickle.load(from_parent) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__ | |
| self._semlock = _multiprocessing.SemLock._rebuild(*state) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58347) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58347) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58345) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58345) exited unexpectedly | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap | |
| self.run() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/process.py", line 108, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 27, in _wrap | |
| sys.exit(1) | |
| SystemExit: 1 | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/rightrec/train.py", line 326, in <module> | |
| main() | |
| File "/private/home/sgross/rightrec/train.py", line 318, in main | |
| torch.multiprocessing.spawn( | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 200, in spawn | |
| return start_processes(fn, args, nprocs, join, daemon, start_method='spawn') | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 158, in start_processes | |
| while not context.join(): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 119, in join | |
| raise Exception(msg) | |
| Exception: | |
| -- Process 1 terminated with the following error: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data | |
| data = self._data_queue.get(timeout=timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/queues.py", line 107, in get | |
| if not self._poll(timeout): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 257, in poll | |
| return self._poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 424, in _poll | |
| r = wait([self], timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/connection.py", line 930, in wait | |
| ready = selector.select(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/selectors.py", line 415, in select | |
| fd_event_list = self._selector.poll(timeout) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler | |
| _error_if_any_worker_fails() | |
| RuntimeError: DataLoader worker (pid 58338) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace. | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap | |
| fn(i, *args) | |
| File "/private/home/sgross/rightrec/train.py", line 236, in train | |
| for inputs, targets in train_loader: | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ | |
| data = self._next_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data | |
| idx, data = self._get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data | |
| success, data = self._try_get_data() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data | |
| raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) | |
| RuntimeError: DataLoader worker (pid(s) 58338) exited unexpectedly | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| Traceback (most recent call last): | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 286, in _run_finalizers | |
| finalizer() | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/util.py", line 205, in __call__ | |
| res = self._callback(*self._args, **self._kwargs) | |
| File "/private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/synchronize.py", line 87, in _cleanup | |
| sem_unlink(name) | |
| FileNotFoundError: [Errno 2] No such file or directory | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 104 leaked semaphore objects to clean up at shutdown | |
| warnings.warn('resource_tracker: There appear to be %d ' | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-i_c5vf76': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-cydyvl4w': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-kgqqwupb': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-8m1lxtn0': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-ee27nftv': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-4klj5ndv': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-4s0h5plm': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-wud9al_o': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-d5ciwdux': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-h7wiji_u': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-aakfp4fv': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-8o1s8943': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-niwqbj9c': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-9nf85d7d': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-2syt7fks': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-dez9oywf': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-7i2wfzt2': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-2akdnrvz': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-fqgjol_b': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-o2yu23nk': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-57l1cx8d': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-qtiruoar': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-48bmkcx3': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-qhrgp40u': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-lv80t6jf': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-bwck76gf': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-9147pm_n': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-mi3twqq2': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-w5wsm_sk': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-a4ieflsa': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-n9xerarz': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-60eg5_w1': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-wftu14uc': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-_2f3rcz8': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-5ofcq7zr': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-ddwa1f7m': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-qn_uyj12': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-wvayrt7s': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-mauo9sh0': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-cazf74b6': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-28fvr5jt': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-vyrgki7_': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-psou2nwd': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-ar99ec7n': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-ob01ml2k': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-722bdo0f': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-4d68tw6t': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-0deki0f_': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-wa7ury_1': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-3jcglm8l': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-yjbl54po': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-7wpvr7te': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-ckxyvg1o': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-pimwrhbj': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-o4ghmld9': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-68jic65t': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-vnuzek0_': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-z8e51wc4': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-phg45jly': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-9unovp60': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-n8o2tu5u': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-60f8ul67': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-q87tm4w9': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-httzg6f2': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-grgfqo6b': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-hl9semjg': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-xb4t_awz': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-3vw9s34e': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-sevp09om': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-djieof_h': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-7r_omihx': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-vymcigu5': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-7fbhwsfa': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-vlilcolw': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-d45nvsl3': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-trfycjnv': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-6fl4r_8u': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-gatw9887': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-big61gcq': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-201pgnus': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-yc2lq0td': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-u_6m0ens': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-c99wplxu': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-dat9gvb9': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-p9h8922r': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-kg0v6e4f': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-ccus0tiy': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-sck2v2eg': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-gs79yyny': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-23tcxlpt': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-al3u5w1e': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-kbc67ww6': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-z83tfu60': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-gmesc_q3': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-u3tghhso': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-57pvcmvf': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-zfv74g_i': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-meod0g4u': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-orxf67mp': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-dyp3sn4r': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-vyorvbhp': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-nvro2pp_': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-gfcsw21g': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| /private/home/sgross/miniconda3/envs/gtn_env2/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-_m_a88vc': [Errno 2] No such file or directory | |
| warnings.warn('resource_tracker: %r: %s' % (name, e)) | |
| srun: error: learnfair5199: task 0: Exited with exit code 1 | |
| srun: Terminating job step 28820487.0 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment