new_error_msg

terminate called after throwing an instance of 'std::runtime_error'
  what():  ../torch/csrc/autograd/profiler_cuda.cpp:36: CUDA initialization error. This can occur if one ru
ns the profiler in CUDA mode on code that creates a DataLoader with num_workers > 0. This operation is curr
ently unsupported; potential workarounds are: (1) don't use the profiler in CUDA mode or (2) use num_worker
s=0 in the DataLoader or (3) Don't profile the data loading portion of your code. https://github.com/pytorc
h/pytorch/issues/6313 tracks profiler support for multi-worker DataLoader.
Traceback (most recent call last):
  File "/scratch/rzou/pt/workspace/torch/utils/data/dataloader.py", line 761, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/scratch/rzou/pt/workspace-env/lib/python3.7/multiprocessing/queues.py", line 104, in get
    if not self._poll(timeout):
  File "/scratch/rzou/pt/workspace-env/lib/python3.7/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/scratch/rzou/pt/workspace-env/lib/python3.7/multiprocessing/connection.py", line 414, in _poll
    r = wait([self], timeout)
  File "/scratch/rzou/pt/workspace-env/lib/python3.7/multiprocessing/connection.py", line 920, in wait
    ready = selector.select(timeout)
  File "/scratch/rzou/pt/workspace-env/lib/python3.7/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
  File "/scratch/rzou/pt/workspace/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 18703) is killed by signal: Aborted.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "mwe.py", line 15, in <module>
    for i, batch in enumerate(data_loader):
  File "/scratch/rzou/pt/workspace/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/scratch/rzou/pt/workspace/torch/utils/data/dataloader.py", line 841, in _next_data
    idx, data = self._get_data()
  File "/scratch/rzou/pt/workspace/torch/utils/data/dataloader.py", line 808, in _get_data
    success, data = self._try_get_data()
  File "/scratch/rzou/pt/workspace/torch/utils/data/dataloader.py", line 774, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 18703) exited unexpectedly