Created
May 1, 2016 19:40
-
-
Save pdeperio/67b715e74ca2c95e89599f374771d222 to your computer and use it in GitHub Desktop.
pax Multicore failure 2
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| nprocs = 4 | |
| discarding /project/lgrandi/anaconda3/envs/pax_head/bin from PATH | |
| prepending /project/lgrandi/anaconda3/envs/pax_head/bin to PATH | |
| Activate: ROOT has been sourced. Environment settings are ready. | |
| ROOTSYS=/project/lgrandi/anaconda3/envs/pax_head | |
| python /project/lgrandi/deployHQ/cax/cax/tasks/process.py 160501_1314 /project/lgrandi/xenon1t/160501_1314 midway-login1 | |
| processor MainProcess L89 INFO This is PAX version 4.9.1, running with configuration for XENON1T. | |
| processing 160501_1314 /project/lgrandi/xenon1t/160501_1314 | |
| ReadZipped MainProcess L101 INFO InputFromFolder: Selecting file /project/lgrandi/xenon1t/160501_1314/XENON1T-0-000000000-000000999-000001000.zip (number 1/92 in folder) for reading | |
| Status: normal. Processing queue: 0 events. Output queue: 0 events. RAM usage: 372.9 (master) 1469.5 (workers) 358.2 (output) | |
| Status: normal. Processing queue: 100 events. Output queue: 0 events. RAM usage: 372.9 (master) 1507.7 (workers) 374.5 (output) | |
| Status: normal. Processing queue: 180 events. Output queue: 0 events. RAM usage: 372.9 (master) 1600.9 (workers) 378.4 (output) | |
| Status: normal. Processing queue: 250 events. Output queue: 0 events. RAM usage: 372.9 (master) 1600.7 (workers) 378.5 (output) | |
| Status: normal. Processing queue: 320 events. Output queue: 0 events. RAM usage: 372.9 (master) 1601.2 (workers) 378.5 (output)#error "You need a ISO C conforming compiler to use the glibc headers" | |
| #error "You need a ISO C conforming compiler to use the glibc headers" | |
| #error "You need a ISO C conforming compiler to use the glibc headers" | |
| #error "You need a ISO C conforming compiler to use the glibc headers" | |
| #error "You need a ISO C conforming compiler to use the glibc headers" | |
| *** Interpreter error recovered *** | |
| *** Interpreter error recovered *** | |
| *** Interpreter error recovered *** | |
| *** Interpreter error recovered *** | |
| *** Interpreter error recovered *** | |
| CheckBounds Process-2 L114 WARNING [Event 0] Pulse 24 in channel 18 (29979-30080) is partially outside event bounds (0-29999). See issue #43 | |
| CheckBounds Process-2 L114 WARNING [Event 0] Pulse 115 in channel 58 (29932-30033) is partially outside event bounds (0-29999). See issue #43 | |
| CheckBounds Process-2 L114 WARNING [Event 0] Pulse 218 in channel 134 (29900-30001) is partially outside event bounds (0-29999). See issue #43 | |
| CheckBounds Process-5 L114 WARNING [Event 10] Pulse 3 in channel 4 (29981-30082) is partially outside event bounds (0-29999). See issue #43 | |
| CheckBounds Process-3 L99 WARNING [Event 20] Pulse 87 in channel 41 (52145-52246) is entirely outside event bounds (0-52144)! See issue #43. | |
| CheckBounds Process-3 L114 WARNING [Event 20] Pulse 194 in channel 74 (52102-52203) is partially outside event bounds (0-52144). See issue #43 | |
| CheckBounds Process-3 L114 WARNING [Event 20] Pulse 224 in channel 84 (52044-52145) is partially outside event bounds (0-52144). See issue #43 | |
| CheckBounds Process-3 L114 WARNING [Event 20] Pulse 255 in channel 93 (52068-52169) is partially outside event bounds (0-52144). See issue #43 | |
| Status: normal. Processing queue: 320 events. Output queue: 0 events. RAM usage: 372.9 (master) 1605.6 (workers) 379.9 (output)/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/numpy/lib/shape_base.py:873: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future | |
| return c.reshape(shape_out) | |
| ROOT.TUnixSystem.ACLiC Process-2 L98 INFO creating shared library /project/lgrandi/xenon1t/processing/pax_event_class-db4c70e5cf345ec91b96ab5cd6c1c183_cpp.so | |
| Status: normal. Processing queue: 380 events. Output queue: 0 events. RAM usage: 372.9 (master) 1619.2 (workers) 379.9 (output) | |
| Status: normal. Processing queue: 450 events. Output queue: 0 events. RAM usage: 372.9 (master) 1619.2 (workers) 379.9 (output)Error: rootcint: failed to rename /project/lgrandi/xenon1t/processing/pax_event_class_db4c70e5cf345ec91b96ab5cd6c1c183_cpp_ACLiC_dict.cxx_+_+_+rootcinttmp to /project/lgrandi/xenon1t/processing/pax_event_class_db4c70e5cf345ec91b96ab5cd6c1c183_cpp_ACLiC_dict.cxx in ReplaceBundleInDict() | |
| Error: rootcint: failed to rename /project/lgrandi/xenon1t/processing/pax_event_class_db4c70e5cf345ec91b96ab5cd6c1c183_cpp_ACLiC_dict.cxx_+_+_+rootcinttmp to /project/lgrandi/xenon1t/processing/pax_event_class_db4c70e5cf345ec91b96ab5cd6c1c183_cpp_ACLiC_dict.h in ReplaceBundleInDict() | |
| Status: normal. Processing queue: 520 events. Output queue: 0 events. RAM usage: 372.9 (master) 1619.9 (workers) 379.9 (output)/project/lgrandi/xenon1t/processing/pax_event_class_db4c70e5cf345ec91b96ab5cd6c1c183_cpp_ACLiC_dict.cxx:17:77: fatal error: pax_event_class_db4c70e5cf345ec91b96ab5cd6c1c183_cpp_ACLiC_dict.h: No such file or directory | |
| #include "pax_event_class_db4c70e5cf345ec91b96ab5cd6c1c183_cpp_ACLiC_dict.h" | |
| ^ | |
| compilation terminated. | |
| c++: error: /project/lgrandi/xenon1t/processing/pax_event_class_db4c70e5cf345ec91b96ab5cd6c1c183_cpp_ACLiC_dict.o: No such file or directory | |
| ROOT.ACLiC Process-2 L98 ERROR Compilation failed! | |
| rootpy.stl Process-2 L386 INFO generating dictionary for std::vector<Hit> ... | |
| Status: normal. Processing queue: 610 events. Output queue: 0 events. RAM usage: 372.9 (master) 1619.9 (workers) 379.9 (output)ROOT.TUnixSystem.ACLiC Process-2 L98 INFO creating shared library /home/tunnell/.cache/rootpy/x86_64-53432/dicts/fcdca0b9610c332f.so | |
| Status: normal. Processing queue: 710 events. Output queue: 0 events. RAM usage: 372.9 (master) 1619.9 (workers) 379.9 (output) | |
| Status: normal. Processing queue: 810 events. Output queue: 0 events. RAM usage: 372.9 (master) 1619.9 (workers) 379.9 (output)ReadZipped MainProcess L101 INFO InputFromFolder: Selecting file /project/lgrandi/xenon1t/160501_1314/XENON1T-0-000001000-000001999-000001000.zip (number 2/92 in folder) for reading | |
| Status: normal. Processing queue: 960 events. Output queue: 0 events. RAM usage: 372.9 (master) 1619.9 (workers) 379.9 (output) | |
| Status: normal. Processing queue: 1000 events. Output queue: 0 events. RAM usage: 372.9 (master) 1619.9 (workers) 379.9 (output) | |
| Status: normal. Processing queue: 1000 events. Output queue: 0 events. RAM usage: 372.9 (master) 1619.9 (workers) 379.9 (output) | |
| Status: normal. Processing queue: 1000 events. Output queue: 0 events. RAM usage: 372.9 (master) 1619.9 (workers) 379.9 (output) | |
| Status: normal. Processing queue: 1000 events. Output queue: 0 events. RAM usage: 372.9 (master) 1619.9 (workers) 379.9 (output)/usr/bin/ld: warning: libpcre.so.1, needed by /project/lgrandi/anaconda3/envs/pax_head/lib/./libCore.so, not found (try using -rpath or -rpath-link) | |
| /usr/bin/ld: warning: liblzma.so.5, needed by /project/lgrandi/anaconda3/envs/pax_head/lib/./libCore.so, not found (try using -rpath or -rpath-link) | |
| /usr/bin/ld: warning: libssl.so.1.0.0, needed by /project/lgrandi/anaconda3/envs/pax_head/lib/./libNet.so, not found (try using -rpath or -rpath-link) | |
| /usr/bin/ld: warning: libcrypto.so.1.0.0, needed by /project/lgrandi/anaconda3/envs/pax_head/lib/./libNet.so, not found (try using -rpath or -rpath-link) | |
| /usr/lib/../lib64/crt1.o: In function `_start': | |
| (.text+0x20): undefined reference to `main' | |
| /home/tunnell/.cache/rootpy/x86_64-53432/dicts/fcdca0b9610c332f_ACLiC_dict.o: In function `Hit::Hit(Hit const&)': | |
| fcdca0b9610c332f_ACLiC_dict.cxx:(.text._ZN3HitC2ERKS_[_ZN3HitC5ERKS_]+0x2a): undefined reference to `vtable for Hit' | |
| /home/tunnell/.cache/rootpy/x86_64-53432/dicts/fcdca0b9610c332f_ACLiC_dict.o: In function `Hit::~Hit()': | |
| fcdca0b9610c332f_ACLiC_dict.cxx:(.text._ZN3HitD2Ev[_ZN3HitD5Ev]+0x13): undefined reference to `vtable for Hit' | |
| /home/tunnell/.cache/rootpy/x86_64-53432/dicts/fcdca0b9610c332f_ACLiC_dict.o: In function `Hit::Hit()': | |
| fcdca0b9610c332f_ACLiC_dict.cxx:(.text._ZN3HitC2Ev[_ZN3HitC5Ev]+0x1f): undefined reference to `vtable for Hit' | |
| /home/tunnell/.cache/rootpy/x86_64-53432/dicts/fcdca0b9610c332f_ACLiC_dict.o: In function `Hit::Hit(Hit&&)': | |
| fcdca0b9610c332f_ACLiC_dict.cxx:(.text._ZN3HitC2EOS_[_ZN3HitC5EOS_]+0x2a): undefined reference to `vtable for Hit' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libCore.so: undefined reference to `pcre_exec' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libCore.so: undefined reference to `pcre_study' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libNet.so: undefined reference to `EVP_sha1' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libNet.so: undefined reference to `SSL_CTX_new' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libNet.so: undefined reference to `SSL_set_fd' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libNet.so: undefined reference to `SSL_CTX_use_certificate_chain_file' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libNet.so: undefined reference to `SSL_shutdown' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libCore.so: undefined reference to `lzma_code' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libNet.so: undefined reference to `SSL_set_quiet_shutdown' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libNet.so: undefined reference to `SSL_get_error' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libCore.so: undefined reference to `lzma_end' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libNet.so: undefined reference to `SSL_CTX_load_verify_locations' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libNet.so: undefined reference to `SSL_CTX_use_PrivateKey_file' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libNet.so: undefined reference to `SSLv23_method' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libNet.so: undefined reference to `SSL_peek' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libNet.so: undefined reference to `SSL_read' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libCore.so: undefined reference to `pcre_free' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libNet.so: undefined reference to `SSL_library_init' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libNet.so: undefined reference to `SSL_connect' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libNet.so: undefined reference to `SSL_write' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libNet.so: undefined reference to `SSL_free' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libNet.so: undefined reference to `SSL_new' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libNet.so: undefined reference to `HMAC' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libNet.so: undefined reference to `SSL_CTX_free' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libCore.so: undefined reference to `lzma_easy_encoder' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libCore.so: undefined reference to `lzma_stream_decoder' | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/./libCore.so: undefined reference to `pcre_compile' | |
| collect2: error: ld returned 1 exit status | |
| Process Process-2: | |
| Traceback (most recent call last): | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/process.py", line 254, in _bootstrap | |
| self.run() | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/process.py", line 93, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 257, in __init__ | |
| self.run() | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 436, in run | |
| event_block[i] = self.process_event(event) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 361, in process_event | |
| event = plugin.process_event(event) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/plugin.py", line 90, in process_event | |
| event = self._process_event(event) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/plugin.py", line 107, in _process_event | |
| return self.transform_event(event) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/plugins/io/ROOTClass.py", line 68, in transform_event | |
| other_process_compiles=(self.processor.worker_id not in ['master', 'processing_0'])) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/plugins/io/ROOTClass.py", line 388, in load_event_class_code | |
| load_event_class(os.path.abspath(class_filename)) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/plugins/io/ROOTClass.py", line 408, in load_event_class | |
| stl.generate("std::vector<%s>" % name, ['<vector>', "%s" % filename], True) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/rootpy/stl.py", line 402, in generate | |
| sourcepath, 'k-', libname, DICTS_PATH) != 1): | |
| RuntimeError: (file "/home/tunnell/.cache/rootpy/x86_64-53432/dicts/fcdca0b9610c332f.so", line -1) Failed to load Dynamic link library /home/tunnell/.cache/rootpy/x86_64-53432/dicts/fcdca0b9610c332f.so | |
| processor MainProcess L524 CRITICAL Crash detected, giving worker processes five seconds to die in peace | |
| processor MainProcess L526 CRITICAL That's it, farewell cruel world! | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/numpy/lib/shape_base.py:873: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future | |
| return c.reshape(shape_out) | |
| ROOTClass_helpers Process-5 L380 INFO Compiled pax event class has been found, sleeping 20 sec to ensure compilation and dictionary generation have finished | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/numpy/lib/shape_base.py:873: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future | |
| return c.reshape(shape_out) | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/numpy/lib/nanfunctions.py:227: RuntimeWarning: All-NaN axis encountered | |
| warnings.warn("All-NaN axis encountered", RuntimeWarning) | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/numpy/lib/nanfunctions.py:326: RuntimeWarning: All-NaN slice encountered | |
| warnings.warn("All-NaN slice encountered", RuntimeWarning) | |
| ROOTClass_helpers Process-3 L380 INFO Compiled pax event class has been found, sleeping 20 sec to ensure compilation and dictionary generation have finished | |
| /project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/numpy/lib/shape_base.py:873: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future | |
| return c.reshape(shape_out) | |
| ROOTClass_helpers Process-4 L380 INFO Compiled pax event class has been found, sleeping 20 sec to ensure compilation and dictionary generation have finished | |
| rootpy.stl Process-4 L386 INFO generating dictionary for std::vector<Peak> ... | |
| ROOT.TUnixSystem.ACLiC Process-4 L98 INFO creating shared library /home/tunnell/.cache/rootpy/x86_64-53432/dicts/0595b0eccdd0bd46.so | |
| rootpy.stl Process-4 L386 INFO generating dictionary for std::vector<Pulse> ... | |
| ROOT.TUnixSystem.ACLiC Process-4 L98 INFO creating shared library /home/tunnell/.cache/rootpy/x86_64-53432/dicts/8482689194eafed6.so | |
| ROOT.TUnixSystem.DispatchSignals Process-4 L98 ERROR write on a pipe with no one to read it | |
| stack Process-4 L40 ERROR File "/project/lgrandi/deployHQ/cax/cax/tasks/process.py", line 151, in <module> | |
| stack Process-4 L40 ERROR events = self.run_doc.get('trigger', {}).get('events_built', 0) | |
| stack Process-4 L40 ERROR File "/project/lgrandi/deployHQ/cax/cax/tasks/process.py", line 63, in process | |
| stack Process-4 L40 ERROR else: | |
| stack Process-4 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 130, in __init__ | |
| stack Process-4 L40 ERROR w.start() | |
| stack Process-4 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/process.py", line 105, in start | |
| stack Process-4 L40 ERROR self._popen = self._Popen(self) | |
| stack Process-4 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/context.py", line 212, in _Popen | |
| stack Process-4 L40 ERROR return _default_context.get_context().Process._Popen(process_obj) | |
| stack Process-4 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/context.py", line 267, in _Popen | |
| stack Process-4 L40 ERROR return Popen(process_obj) | |
| stack Process-4 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/popen_fork.py", line 21, in __init__ | |
| stack Process-4 L40 ERROR self._launch(process_obj) | |
| stack Process-4 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/popen_fork.py", line 77, in _launch | |
| stack Process-4 L40 ERROR code = process_obj._bootstrap() | |
| stack Process-4 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/process.py", line 254, in _bootstrap | |
| stack Process-4 L40 ERROR self.run() | |
| stack Process-4 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/process.py", line 93, in run | |
| stack Process-4 L40 ERROR self._target(*self._args, **self._kwargs) | |
| stack Process-4 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 257, in __init__ | |
| stack Process-4 L40 ERROR self.run() | |
| stack Process-4 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 435, in run | |
| stack Process-4 L40 ERROR self.check_crash() | |
| stack Process-4 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 522, in check_crash | |
| stack Process-4 L40 ERROR if self.status.value == MP_STATUS['crashing']: | |
| stack Process-4 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/managers.py", line 1047, in get | |
| stack Process-4 L40 ERROR return self._callmethod('get') | |
| stack Process-4 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/managers.py", line 731, in _callmethod | |
| stack Process-4 L40 ERROR conn.send((self._id, methodname, args, kwds)) | |
| stack Process-4 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 206, in send | |
| stack Process-4 L40 ERROR self._send_bytes(ForkingPickler.dumps(obj)) | |
| stack Process-4 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 413, in _send_bytes | |
| stack Process-4 L40 ERROR self._send(chunk) | |
| stack Process-4 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 369, in _send | |
| stack Process-4 L40 ERROR n = write(self._handle, buf) | |
| ROOT.TUnixSystem.DispatchSignals Process-4 L98 ERROR write on a pipe with no one to read it | |
| Process Process-4: | |
| Traceback (most recent call last): | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 435, in run | |
| self.check_crash() | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 522, in check_crash | |
| if self.status.value == MP_STATUS['crashing']: | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/managers.py", line 1047, in get | |
| return self._callmethod('get') | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/managers.py", line 731, in _callmethod | |
| conn.send((self._id, methodname, args, kwds)) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 206, in send | |
| self._send_bytes(ForkingPickler.dumps(obj)) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 413, in _send_bytes | |
| self._send(chunk) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 369, in _send | |
| n = write(self._handle, buf) | |
| BrokenPipeError: [Errno 32] Broken pipe | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/process.py", line 254, in _bootstrap | |
| self.run() | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/process.py", line 93, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 257, in __init__ | |
| self.run() | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 439, in run | |
| self.status.value = MP_STATUS['crashing'] | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/managers.py", line 1049, in set | |
| return self._callmethod('set', (value,)) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/managers.py", line 731, in _callmethod | |
| conn.send((self._id, methodname, args, kwds)) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 206, in send | |
| self._send_bytes(ForkingPickler.dumps(obj)) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 413, in _send_bytes | |
| self._send(chunk) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 369, in _send | |
| n = write(self._handle, buf) | |
| BrokenPipeError: [Errno 32] Broken pipe | |
| rootpy.stl Process-5 L386 INFO generating dictionary for std::vector<Peak> ... | |
| ROOT.TUnixSystem.ACLiC Process-5 L98 INFO creating shared library /home/tunnell/.cache/rootpy/x86_64-53432/dicts/0595b0eccdd0bd46.so | |
| rootpy.stl Process-3 L386 INFO generating dictionary for std::vector<Peak> ... | |
| ROOT.TUnixSystem.DispatchSignals Process-5 L98 ERROR write on a pipe with no one to read it | |
| stack Process-5 L40 ERROR File "/project/lgrandi/deployHQ/cax/cax/tasks/process.py", line 151, in <module> | |
| stack Process-5 L40 ERROR events = self.run_doc.get('trigger', {}).get('events_built', 0) | |
| stack Process-5 L40 ERROR File "/project/lgrandi/deployHQ/cax/cax/tasks/process.py", line 63, in process | |
| stack Process-5 L40 ERROR else: | |
| stack Process-5 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 130, in __init__ | |
| stack Process-5 L40 ERROR w.start() | |
| stack Process-5 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/process.py", line 105, in start | |
| stack Process-5 L40 ERROR self._popen = self._Popen(self) | |
| stack Process-5 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/context.py", line 212, in _Popen | |
| stack Process-5 L40 ERROR return _default_context.get_context().Process._Popen(process_obj) | |
| stack Process-5 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/context.py", line 267, in _Popen | |
| stack Process-5 L40 ERROR return Popen(process_obj) | |
| stack Process-5 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/popen_fork.py", line 21, in __init__ | |
| stack Process-5 L40 ERROR self._launch(process_obj) | |
| stack Process-5 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/popen_fork.py", line 77, in _launch | |
| stack Process-5 L40 ERROR code = process_obj._bootstrap() | |
| stack Process-5 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/process.py", line 254, in _bootstrap | |
| stack Process-5 L40 ERROR self.run() | |
| stack Process-5 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/process.py", line 93, in run | |
| stack Process-5 L40 ERROR self._target(*self._args, **self._kwargs) | |
| stack Process-5 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 257, in __init__ | |
| stack Process-5 L40 ERROR self.run() | |
| stack Process-5 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 435, in run | |
| stack Process-5 L40 ERROR self.check_crash() | |
| stack Process-5 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 522, in check_crash | |
| stack Process-5 L40 ERROR if self.status.value == MP_STATUS['crashing']: | |
| stack Process-5 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/managers.py", line 1047, in get | |
| stack Process-5 L40 ERROR return self._callmethod('get') | |
| stack Process-5 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/managers.py", line 731, in _callmethod | |
| stack Process-5 L40 ERROR conn.send((self._id, methodname, args, kwds)) | |
| stack Process-5 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 206, in send | |
| stack Process-5 L40 ERROR self._send_bytes(ForkingPickler.dumps(obj)) | |
| stack Process-5 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 413, in _send_bytes | |
| stack Process-5 L40 ERROR self._send(chunk) | |
| stack Process-5 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 369, in _send | |
| stack Process-5 L40 ERROR n = write(self._handle, buf) | |
| ROOT.TUnixSystem.DispatchSignals Process-5 L98 ERROR write on a pipe with no one to read it | |
| Process Process-5: | |
| Traceback (most recent call last): | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 435, in run | |
| self.check_crash() | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 522, in check_crash | |
| if self.status.value == MP_STATUS['crashing']: | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/managers.py", line 1047, in get | |
| return self._callmethod('get') | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/managers.py", line 731, in _callmethod | |
| conn.send((self._id, methodname, args, kwds)) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 206, in send | |
| self._send_bytes(ForkingPickler.dumps(obj)) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 413, in _send_bytes | |
| self._send(chunk) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 369, in _send | |
| n = write(self._handle, buf) | |
| BrokenPipeError: [Errno 32] Broken pipe | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/process.py", line 254, in _bootstrap | |
| self.run() | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/process.py", line 93, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 257, in __init__ | |
| self.run() | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 439, in run | |
| self.status.value = MP_STATUS['crashing'] | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/managers.py", line 1049, in set | |
| return self._callmethod('set', (value,)) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/managers.py", line 731, in _callmethod | |
| conn.send((self._id, methodname, args, kwds)) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 206, in send | |
| self._send_bytes(ForkingPickler.dumps(obj)) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 413, in _send_bytes | |
| self._send(chunk) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 369, in _send | |
| n = write(self._handle, buf) | |
| BrokenPipeError: [Errno 32] Broken pipe | |
| ROOT.TUnixSystem.DispatchSignals Process-3 L98 ERROR write on a pipe with no one to read it | |
| stack Process-3 L40 ERROR File "/project/lgrandi/deployHQ/cax/cax/tasks/process.py", line 151, in <module> | |
| stack Process-3 L40 ERROR events = self.run_doc.get('trigger', {}).get('events_built', 0) | |
| stack Process-3 L40 ERROR File "/project/lgrandi/deployHQ/cax/cax/tasks/process.py", line 63, in process | |
| stack Process-3 L40 ERROR else: | |
| stack Process-3 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 130, in __init__ | |
| stack Process-3 L40 ERROR w.start() | |
| stack Process-3 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/process.py", line 105, in start | |
| stack Process-3 L40 ERROR self._popen = self._Popen(self) | |
| stack Process-3 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/context.py", line 212, in _Popen | |
| stack Process-3 L40 ERROR return _default_context.get_context().Process._Popen(process_obj) | |
| stack Process-3 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/context.py", line 267, in _Popen | |
| stack Process-3 L40 ERROR return Popen(process_obj) | |
| stack Process-3 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/popen_fork.py", line 21, in __init__ | |
| stack Process-3 L40 ERROR self._launch(process_obj) | |
| stack Process-3 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/popen_fork.py", line 77, in _launch | |
| stack Process-3 L40 ERROR code = process_obj._bootstrap() | |
| stack Process-3 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/process.py", line 254, in _bootstrap | |
| stack Process-3 L40 ERROR self.run() | |
| stack Process-3 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/process.py", line 93, in run | |
| stack Process-3 L40 ERROR self._target(*self._args, **self._kwargs) | |
| stack Process-3 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 257, in __init__ | |
| stack Process-3 L40 ERROR self.run() | |
| stack Process-3 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 435, in run | |
| stack Process-3 L40 ERROR self.check_crash() | |
| stack Process-3 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 522, in check_crash | |
| stack Process-3 L40 ERROR if self.status.value == MP_STATUS['crashing']: | |
| stack Process-3 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/managers.py", line 1047, in get | |
| stack Process-3 L40 ERROR return self._callmethod('get') | |
| stack Process-3 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/managers.py", line 731, in _callmethod | |
| stack Process-3 L40 ERROR conn.send((self._id, methodname, args, kwds)) | |
| stack Process-3 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 206, in send | |
| stack Process-3 L40 ERROR self._send_bytes(ForkingPickler.dumps(obj)) | |
| stack Process-3 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 413, in _send_bytes | |
| stack Process-3 L40 ERROR self._send(chunk) | |
| stack Process-3 L40 ERROR File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 369, in _send | |
| stack Process-3 L40 ERROR n = write(self._handle, buf) | |
| ROOT.TUnixSystem.DispatchSignals Process-3 L98 ERROR write on a pipe with no one to read it | |
| Process Process-3: | |
| Traceback (most recent call last): | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 435, in run | |
| self.check_crash() | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 522, in check_crash | |
| if self.status.value == MP_STATUS['crashing']: | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/managers.py", line 1047, in get | |
| return self._callmethod('get') | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/managers.py", line 731, in _callmethod | |
| conn.send((self._id, methodname, args, kwds)) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 206, in send | |
| self._send_bytes(ForkingPickler.dumps(obj)) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 413, in _send_bytes | |
| self._send(chunk) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 369, in _send | |
| n = write(self._handle, buf) | |
| BrokenPipeError: [Errno 32] Broken pipe | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/process.py", line 254, in _bootstrap | |
| self.run() | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/process.py", line 93, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 257, in __init__ | |
| self.run() | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 439, in run | |
| self.status.value = MP_STATUS['crashing'] | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/managers.py", line 1049, in set | |
| return self._callmethod('set', (value,)) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/managers.py", line 731, in _callmethod | |
| conn.send((self._id, methodname, args, kwds)) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 206, in send | |
| self._send_bytes(ForkingPickler.dumps(obj)) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 413, in _send_bytes | |
| self._send(chunk) | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/multiprocessing/connection.py", line 369, in _send | |
| n = write(self._handle, buf) | |
| BrokenPipeError: [Errno 32] Broken pipe | |
| Traceback (most recent call last): | |
| File "/project/lgrandi/deployHQ/cax/cax/tasks/process.py", line 151, in <module> | |
| process(sys.argv[1], sys.argv[2], sys.argv[3]) | |
| File "/project/lgrandi/deployHQ/cax/cax/tasks/process.py", line 64, in process | |
| p.run() | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 474, in run | |
| self.master_heartbeat() | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 518, in master_heartbeat | |
| self.check_crash() | |
| File "/project/lgrandi/anaconda3/envs/pax_head/lib/python3.4/site-packages/pax-4.9.1-py3.4.egg/pax/core.py", line 527, in check_crash | |
| raise RuntimeError("Terminated pax multiprocessing due to crash in one of the workers.") | |
| RuntimeError: Terminated pax multiprocessing due to crash in one of the workers. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment