plasma_demo_tb.py

/Users/runner/work/crossbow/crossbow/arrow/cpp/src/plasma/store.cc:1274: Allowing the Plasma store to use up to 214.748GB of memory.
/Users/runner/work/crossbow/crossbow/arrow/cpp/src/plasma/store.cc:1297: Starting object store with directory /tmp and huge page support disabled
pid: 90940, id: ObjectID(5bfe481fec7b0af28e49a7d0fd0e95bfc9fda702), lock: True: PlasmaObjectExists
pid: 90940, id: ObjectID(9c4040a5d038b77cbff1218571ca01f7ff8cc8f7), lock: True: PUT
/Users/runner/work/crossbow/crossbow/arrow/cpp/src/plasma/store.cc:754: Disconnecting client on fd 7
pid: 90940, id: ObjectID(5bfe481fec7b0af28e49a7d0fd0e95bfc9fda702), lock: True: GET
Traceback (most recent call last):
  File "plasma_demo.py", line 148, in <module>
    main()
  File "plasma_demo.py", line 143, in main
    train(args.num_workers)
  File "plasma_demo.py", line 130, in train
    for batch in train_dl:
  File "/Users/sshleifer/miniconda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 352, in __iter__
    return self._get_iterator()
  File "/Users/sshleifer/miniconda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 294, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/Users/sshleifer/miniconda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 827, in __init__
    self._reset(loader, first_iter=True)
  File "/Users/sshleifer/miniconda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 857, in _reset
    self._try_put_index()
  File "/Users/sshleifer/miniconda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1091, in _try_put_index
    index = self._next_index()
  File "/Users/sshleifer/miniconda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 427, in _next_index
    return next(self._sampler_iter)  # may raise StopIteration
  File "/Users/sshleifer/miniconda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 227, in __iter__
    for idx in self.sampler:
  File "/Users/sshleifer/miniconda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 67, in __iter__
    return iter(range(len(self.data_source)))
  File "plasma_demo.py", line 120, in __len__
    return len(self.pv)
  File "plasma_demo.py", line 97, in __len__
    self._n = len(self.array)
  File "plasma_demo.py", line 46, in array
    ret = self.client.get(self.object_id)
  File "pyarrow/_plasma.pyx", line 595, in pyarrow._plasma.PlasmaClient.get
  File "pyarrow/_plasma.pyx", line 583, in pyarrow._plasma.PlasmaClient.get
  File "pyarrow/_plasma.pyx", line 431, in pyarrow._plasma.PlasmaClient.get_buffers
  File "pyarrow/_plasma.pyx", line 325, in pyarrow._plasma.PlasmaClient._get_object_buffers
  File "pyarrow/_plasma.pyx", line 289, in pyarrow._plasma.plasma_check_status
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
OSError: Bad file descriptor
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/Users/sshleifer/miniconda/lib/python3.8/multiprocessing/popen_fork.py", line 27, in poll
    pid, sts = os.waitpid(self.pid, flag)
  File "/Users/sshleifer/miniconda/lib/python3.8/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 90945) is killed by signal: Unknown signal: 0.
/Users/runner/work/crossbow/crossbow/arrow/cpp/src/plasma/store.cc:754: Disconnecting client on fd 6
/Users/runner/work/crossbow/crossbow/arrow/cpp/src/plasma/store.cc:754: Disconnecting client on fd 6