Skip to content

Instantly share code, notes, and snippets.

@beojan

beojan/err.txt Secret

Created March 22, 2021 13:43
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save beojan/3b7240216ee846f7d5a374484eda4dd0 to your computer and use it in GitHub Desktop.
Save beojan/3b7240216ee846f7d5a374484eda4dd0 to your computer and use it in GitHub Desktop.
2021-03-22 13:42:07,325 | packtivity.asyncback | INFO | configured pool size to 8
2021-03-22 13:42:07,478 | yadage.creators | INFO | initializing workflow with initdata: {'dxaod_files_mc16e': ['/eos/project/r/recast/atlas/ANA-HDBS-2018-41/mc/scalar/mc16_13TeV.450253.MadGraphHerwig7EvtGen_NNPDF23LO_X400tohh_bbbb.deriv.DAOD_EXOT8.e7352_s3126_r10724_p4089/*'], 'dxaod_files_mc16d': ['/eos/project/r/recast/atlas/ANA-HDBS-2018-41/mc/scalar/mc16_13TeV.450253.MadGraphHerwig7EvtGen_NNPDF23LO_X400tohh_bbbb.deriv.DAOD_EXOT8.e7352_s3126_r10201_p4089/*'], 'data_dir': '/eos/project/r/recast/atlas/ANA-HDBS-2018-41/data/NNT_FEB21/NNT_FEB21', 'dxaod_files_mc16a': ['/eos/project/r/recast/atlas/ANA-HDBS-2018-41/mc/scalar/mc16_13TeV.450253.MadGraphHerwig7EvtGen_NNPDF23LO_X400tohh_bbbb.deriv.DAOD_EXOT8.e7352_s3126_r9364_p4089/*'], 'did': 450253, 'xsec_in_pb': 114.117} discover: True relative: True
2021-03-22 13:42:07,479 | adage.pollingexec | INFO | preparing adage coroutine.
2021-03-22 13:42:07,479 | adage | INFO | starting state loop.
2021-03-22 13:42:07,619 | yadage.wflowview | INFO | added </init:0|defined|unknown>
2021-03-22 13:42:08,061 | yadage.wflowview | INFO | added </mini:0|defined|unknown>
2021-03-22 13:42:08,641 | yadage.wflowview | INFO | added </resolved_nnt:0|defined|unknown>
2021-03-22 13:42:09,174 | yadage.wflowview | INFO | added </limit:0|defined|unknown>
2021-03-22 13:42:09,550 | adage.pollingexec | INFO | submitting nodes [</init:0|defined|known>]
2021-03-22 13:42:09,671 | pack.init.step | INFO | publishing data: <TypedLeafs: {u'dxaod_files_mc16e': [u'/eos/project/r/recast/atlas/ANA-HDBS-2018-41/mc/scalar/mc16_13TeV.450253.MadGraphHerwig7EvtGen_NNPDF23LO_X400tohh_bbbb.deriv.DAOD_EXOT8.e7352_s3126_r10724_p4089/*'], u'dxaod_files_mc16d': [u'/eos/project/r/recast/atlas/ANA-HDBS-2018-41/mc/scalar/mc16_13TeV.450253.MadGraphHerwig7EvtGen_NNPDF23LO_X400tohh_bbbb.deriv.DAOD_EXOT8.e7352_s3126_r10201_p4089/*'], u'data_dir': u'/eos/project/r/recast/atlas/ANA-HDBS-2018-41/data/NNT_FEB21/NNT_FEB21', u'dxaod_files_mc16a': [u'/eos/project/r/recast/atlas/ANA-HDBS-2018-41/mc/scalar/mc16_13TeV.450253.MadGraphHerwig7EvtGen_NNPDF23LO_X400tohh_bbbb.deriv.DAOD_EXOT8.e7352_s3126_r9364_p4089/*'], u'did': 450253, u'xsec_in_pb': 114.117}>
2021-03-22 13:42:09,672 | adage | INFO | unsubmittable: 0 | submitted: 0 | successful: 0 | failed: 0 | total: 4 | open rules: 0 | applied rules: 4
2021-03-22 13:42:10,064 | adage.node | INFO | node ready </init:0|success|known>
2021-03-22 13:42:10,064 | adage.pollingexec | INFO | submitting nodes [</mini:0|defined|known>]
2021-03-22 13:42:10,065 | pack.mini.step | INFO | starting file logging for topic: step
2021-03-22 13:42:11,184 | pack.mini.step | ERROR | non-zero return code raising exception
2021-03-22 13:42:11,185 | pack.mini.step | ERROR | subprocess failed. code: 1, command docker pull gitlab-registry.cern.ch/hh4b/xhhcommon:21.2.123.FEB21-0-9653efae
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/packtivity/handlers/execution_handlers.py", line 331, in execute_and_tail_subprocess
returncode=proc.returncode, cmd=command_string
CalledProcessError: Command 'docker pull gitlab-registry.cern.ch/hh4b/xhhcommon:21.2.123.FEB21-0-9653efae' returned non-zero exit status 1
2021-03-22 13:42:11,185 | pack.mini.step | ERROR | job execution if job {'wflow_hints': {'is_purepub': False}, 'name': u'mini', 'wflow_stage_node_idx': 0, 'wflow_offset': u'', 'wflow_stage': u'mini', 'wflow_node_id': '6bc3adfb-78b2-4b37-90d5-f0b80319d83d'} raise exception exception
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/packtivity/syncbackends.py", line 187, in run_packtivity
run_in_env(job, env, state, metadata, pack_config, exec_config)
File "/usr/lib/python2.7/site-packages/packtivity/syncbackends.py", line 125, in run_in_env
return handler(exec_config, environment, state, job, metadata)
File "/usr/lib/python2.7/site-packages/packtivity/handlers/execution_handlers.py", line 448, in docker_enc_handler
result = run(config, state, log, metadata, rspec)
File "/usr/lib/python2.7/site-packages/packtivity/handlers/execution_handlers.py", line 387, in run_containers_in_docker_runtime
logging_topic="pull",
File "/usr/lib/python2.7/site-packages/packtivity/handlers/execution_handlers.py", line 338, in execute_and_tail_subprocess
raise RuntimeError("failed container execution subprocess. %s", command_string)
RuntimeError: ('failed container execution subprocess. %s', 'docker pull gitlab-registry.cern.ch/hh4b/xhhcommon:21.2.123.FEB21-0-9653efae')
2021-03-22 13:42:11,675 | adage.node | INFO | node ready </mini:0|failed|known>
2021-03-22 13:42:11,696 | adage.controllerutil | INFO | no nodes can be run anymore and no rules are applicable
2021-03-22 13:42:11,696 | adage.controllerutil | INFO | no nodes can be run anymore and no rules are applicable
2021-03-22 13:42:11,696 | adage | ERROR | some weird exception caught in adage process loop
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/adage/__init__.py", line 51, in run_polling_workflow
for stepnum, controller in enumerate(coroutine):
File "/usr/lib/python2.7/site-packages/adage/pollingexec.py", line 89, in adage_coroutine
raise RuntimeError('workflow finished but failed')
RuntimeError: workflow finished but failed
2021-03-22 13:42:11,696 | adage | ERROR | node: </mini:0|failed|known> failed. reason: unknown
2021-03-22 13:42:11,696 | adage | INFO | unsubmittable: 2 | submitted: 0 | successful: 1 | failed: 1 | total: 4 | open rules: 0 | applied rules: 4
2021-03-22 13:42:16,577 | yadage.steering_api | INFO | done. dumping workflow to disk.
2021-03-22 13:42:16,582 | yadage.steering | ERROR | workflow failed
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yadage/steering.py", line 39, in from_file
steering_api.run_workflow(**data)
File "/usr/lib/python2.7/site-packages/yadage/steering_api.py", line 20, in run_workflow
pass
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/usr/lib/python2.7/site-packages/yadage/steering_api.py", line 118, in steering_ctx
cache=cache,
File "/usr/lib/python2.7/site-packages/yadage/steering_api.py", line 60, in execute_steering
ys.run_adage(backend)
File "/usr/lib/python2.7/site-packages/yadage/steering_object.py", line 100, in run_adage
adage.rundag(controller=self.controller, **self.adage_kwargs)
File "/usr/lib/python2.7/site-packages/adage/__init__.py", line 137, in rundag
run_polling_workflow(controller, coroutine, update_interval, trackerlist, maxsteps)
File "/usr/lib/python2.7/site-packages/adage/__init__.py", line 51, in run_polling_workflow
for stepnum, controller in enumerate(coroutine):
File "/usr/lib/python2.7/site-packages/adage/pollingexec.py", line 89, in adage_coroutine
raise RuntimeError('workflow finished but failed')
RuntimeError: workflow finished but failed
Error: Workflow failed
Traceback (most recent call last):
File "/home/beojan/.local/bin/recast", line 8, in <module>
sys.exit(recastatlas())
File "/usr/lib/python3.9/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3.9/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3.9/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/beojan/.local/lib/python3.9/site-packages/recastatlas/subcommands/run.py", line 50, in run
run_sync(name, spec, backend=backend)
File "/home/beojan/.local/lib/python3.9/site-packages/recastatlas/backends/__init__.py", line 191, in run_sync
subprocess.check_call(command)
File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker', 'run', '--rm', '-i', '-v', '/home/beojan/DPhil/hh4b/Recast/recast-resonant:/home/beojan/DPhil/hh4b/Recast/recast-resonant', '-w', '/home/beojan/DPhil/hh4b/Recast/recast-resonant', '-v', '/var/run/docker.sock:/var/run/docker.sock', '-e', 'PACKTIVITY_CVMFS_LOCATION=/cvmfs', '-e', 'PACKTIVITY_CVMFS_PROPAGATION=rprivate', '-e', 'PACKTIVITY_AUTH_LOCATION=None', '-e', 'YADAGE_SCHEMA_LOAD_TOKEN=None', 'recast/recastatlas:v0.1.0', 'sh', '-c', 'mkdir -p ~/.docker\necho \'{}\' > ~/.docker/config.json \ncat << \'EOF\' | yadage-run -f - \n{"dataarg": "recast-641382ce", "dataopts": {}, "initdata": {"did": 450253, "xsec_in_pb": 114.117, "dxaod_files_mc16a": ["/eos/project/r/recast/atlas/ANA-HDBS-2018-41/mc/scalar/mc16_13TeV.450253.MadGraphHerwig7EvtGen_NNPDF23LO_X400tohh_bbbb.deriv.DAOD_EXOT8.e7352_s3126_r9364_p4089/*"], "dxaod_files_mc16d": ["/eos/project/r/recast/atlas/ANA-HDBS-2018-41/mc/scalar/mc16_13TeV.450253.MadGraphHerwig7EvtGen_NNPDF23LO_X400tohh_bbbb.deriv.DAOD_EXOT8.e7352_s3126_r10201_p4089/*"], "dxaod_files_mc16e": ["/eos/project/r/recast/atlas/ANA-HDBS-2018-41/mc/scalar/mc16_13TeV.450253.MadGraphHerwig7EvtGen_NNPDF23LO_X400tohh_bbbb.deriv.DAOD_EXOT8.e7352_s3126_r10724_p4089/*"], "data_dir": "/eos/project/r/recast/atlas/ANA-HDBS-2018-41/data/NNT_FEB21/NNT_FEB21"}, "workflow": "workflow.yml", "toplevel": "/home/beojan/DPhil/hh4b/Recast/recast-resonant/specs", "visualize": true}\nEOF\n']' returned non-zero exit status 1.
2021-03-22 13:42:07,325 | packtivity.asyncback | INFO | configured pool size to 8
2021-03-22 13:42:07,478 | yadage.creators | INFO | initializing workflow with initdata: {'dxaod_files_mc16e': ['/eos/project/r/recast/atlas/ANA-HDBS-2018-41/mc/scalar/mc16_13TeV.450253.MadGraphHerwig7EvtGen_NNPDF23LO_X400tohh_bbbb.deriv.DAOD_EXOT8.e7352_s3126_r10724_p4089/*'], 'dxaod_files_mc16d': ['/eos/project/r/recast/atlas/ANA-HDBS-2018-41/mc/scalar/mc16_13TeV.450253.MadGraphHerwig7EvtGen_NNPDF23LO_X400tohh_bbbb.deriv.DAOD_EXOT8.e7352_s3126_r10201_p4089/*'], 'data_dir': '/eos/project/r/recast/atlas/ANA-HDBS-2018-41/data/NNT_FEB21/NNT_FEB21', 'dxaod_files_mc16a': ['/eos/project/r/recast/atlas/ANA-HDBS-2018-41/mc/scalar/mc16_13TeV.450253.MadGraphHerwig7EvtGen_NNPDF23LO_X400tohh_bbbb.deriv.DAOD_EXOT8.e7352_s3126_r9364_p4089/*'], 'did': 450253, 'xsec_in_pb': 114.117} discover: True relative: True
2021-03-22 13:42:07,479 | adage.pollingexec | INFO | preparing adage coroutine.
2021-03-22 13:42:07,479 | adage | INFO | starting state loop.
2021-03-22 13:42:07,619 | yadage.wflowview | INFO | added </init:0|defined|unknown>
2021-03-22 13:42:08,061 | yadage.wflowview | INFO | added </mini:0|defined|unknown>
2021-03-22 13:42:08,641 | yadage.wflowview | INFO | added </resolved_nnt:0|defined|unknown>
2021-03-22 13:42:09,174 | yadage.wflowview | INFO | added </limit:0|defined|unknown>
2021-03-22 13:42:09,550 | adage.pollingexec | INFO | submitting nodes [</init:0|defined|known>]
2021-03-22 13:42:09,671 | pack.init.step | INFO | publishing data: <TypedLeafs: {u'dxaod_files_mc16e': [u'/eos/project/r/recast/atlas/ANA-HDBS-2018-41/mc/scalar/mc16_13TeV.450253.MadGraphHerwig7EvtGen_NNPDF23LO_X400tohh_bbbb.deriv.DAOD_EXOT8.e7352_s3126_r10724_p4089/*'], u'dxaod_files_mc16d': [u'/eos/project/r/recast/atlas/ANA-HDBS-2018-41/mc/scalar/mc16_13TeV.450253.MadGraphHerwig7EvtGen_NNPDF23LO_X400tohh_bbbb.deriv.DAOD_EXOT8.e7352_s3126_r10201_p4089/*'], u'data_dir': u'/eos/project/r/recast/atlas/ANA-HDBS-2018-41/data/NNT_FEB21/NNT_FEB21', u'dxaod_files_mc16a': [u'/eos/project/r/recast/atlas/ANA-HDBS-2018-41/mc/scalar/mc16_13TeV.450253.MadGraphHerwig7EvtGen_NNPDF23LO_X400tohh_bbbb.deriv.DAOD_EXOT8.e7352_s3126_r9364_p4089/*'], u'did': 450253, u'xsec_in_pb': 114.117}>
2021-03-22 13:42:09,672 | adage | INFO | unsubmittable: 0 | submitted: 0 | successful: 0 | failed: 0 | total: 4 | open rules: 0 | applied rules: 4
2021-03-22 13:42:10,064 | adage.node | INFO | node ready </init:0|success|known>
2021-03-22 13:42:10,064 | adage.pollingexec | INFO | submitting nodes [</mini:0|defined|known>]
2021-03-22 13:42:10,065 | pack.mini.step | INFO | starting file logging for topic: step
2021-03-22 13:42:11,184 | pack.mini.step | ERROR | non-zero return code raising exception
2021-03-22 13:42:11,185 | pack.mini.step | ERROR | subprocess failed. code: 1, command docker pull gitlab-registry.cern.ch/hh4b/xhhcommon:21.2.123.FEB21-0-9653efae
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/packtivity/handlers/execution_handlers.py", line 331, in execute_and_tail_subprocess
returncode=proc.returncode, cmd=command_string
CalledProcessError: Command 'docker pull gitlab-registry.cern.ch/hh4b/xhhcommon:21.2.123.FEB21-0-9653efae' returned non-zero exit status 1
2021-03-22 13:42:11,185 | pack.mini.step | ERROR | job execution if job {'wflow_hints': {'is_purepub': False}, 'name': u'mini', 'wflow_stage_node_idx': 0, 'wflow_offset': u'', 'wflow_stage': u'mini', 'wflow_node_id': '6bc3adfb-78b2-4b37-90d5-f0b80319d83d'} raise exception exception
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/packtivity/syncbackends.py", line 187, in run_packtivity
run_in_env(job, env, state, metadata, pack_config, exec_config)
File "/usr/lib/python2.7/site-packages/packtivity/syncbackends.py", line 125, in run_in_env
return handler(exec_config, environment, state, job, metadata)
File "/usr/lib/python2.7/site-packages/packtivity/handlers/execution_handlers.py", line 448, in docker_enc_handler
result = run(config, state, log, metadata, rspec)
File "/usr/lib/python2.7/site-packages/packtivity/handlers/execution_handlers.py", line 387, in run_containers_in_docker_runtime
logging_topic="pull",
File "/usr/lib/python2.7/site-packages/packtivity/handlers/execution_handlers.py", line 338, in execute_and_tail_subprocess
raise RuntimeError("failed container execution subprocess. %s", command_string)
RuntimeError: ('failed container execution subprocess. %s', 'docker pull gitlab-registry.cern.ch/hh4b/xhhcommon:21.2.123.FEB21-0-9653efae')
2021-03-22 13:42:11,675 | adage.node | INFO | node ready </mini:0|failed|known>
2021-03-22 13:42:11,696 | adage.controllerutil | INFO | no nodes can be run anymore and no rules are applicable
2021-03-22 13:42:11,696 | adage.controllerutil | INFO | no nodes can be run anymore and no rules are applicable
2021-03-22 13:42:11,696 | adage | ERROR | some weird exception caught in adage process loop
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/adage/__init__.py", line 51, in run_polling_workflow
for stepnum, controller in enumerate(coroutine):
File "/usr/lib/python2.7/site-packages/adage/pollingexec.py", line 89, in adage_coroutine
raise RuntimeError('workflow finished but failed')
RuntimeError: workflow finished but failed
2021-03-22 13:42:11,696 | adage | ERROR | node: </mini:0|failed|known> failed. reason: unknown
2021-03-22 13:42:11,696 | adage | INFO | unsubmittable: 2 | submitted: 0 | successful: 1 | failed: 1 | total: 4 | open rules: 0 | applied rules: 4
2021-03-22 13:42:16,577 | yadage.steering_api | INFO | done. dumping workflow to disk.
2021-03-22 13:42:16,582 | yadage.steering | ERROR | workflow failed
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yadage/steering.py", line 39, in from_file
steering_api.run_workflow(**data)
File "/usr/lib/python2.7/site-packages/yadage/steering_api.py", line 20, in run_workflow
pass
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/usr/lib/python2.7/site-packages/yadage/steering_api.py", line 118, in steering_ctx
cache=cache,
File "/usr/lib/python2.7/site-packages/yadage/steering_api.py", line 60, in execute_steering
ys.run_adage(backend)
File "/usr/lib/python2.7/site-packages/yadage/steering_object.py", line 100, in run_adage
adage.rundag(controller=self.controller, **self.adage_kwargs)
File "/usr/lib/python2.7/site-packages/adage/__init__.py", line 137, in rundag
run_polling_workflow(controller, coroutine, update_interval, trackerlist, maxsteps)
File "/usr/lib/python2.7/site-packages/adage/__init__.py", line 51, in run_polling_workflow
for stepnum, controller in enumerate(coroutine):
File "/usr/lib/python2.7/site-packages/adage/pollingexec.py", line 89, in adage_coroutine
raise RuntimeError('workflow finished but failed')
RuntimeError: workflow finished but failed
Error: Workflow failed
Traceback (most recent call last):
File "/home/beojan/.local/bin/recast", line 8, in <module>
sys.exit(recastatlas())
File "/usr/lib/python3.9/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3.9/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3.9/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/beojan/.local/lib/python3.9/site-packages/recastatlas/subcommands/run.py", line 50, in run
run_sync(name, spec, backend=backend)
File "/home/beojan/.local/lib/python3.9/site-packages/recastatlas/backends/__init__.py", line 191, in run_sync
subprocess.check_call(command)
File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker', 'run', '--rm', '-i', '-v', '/home/beojan/DPhil/hh4b/Recast/recast-resonant:/home/beojan/DPhil/hh4b/Recast/recast-resonant', '-w', '/home/beojan/DPhil/hh4b/Recast/recast-resonant', '-v', '/var/run/docker.sock:/var/run/docker.sock', '-e', 'PACKTIVITY_CVMFS_LOCATION=/cvmfs', '-e', 'PACKTIVITY_CVMFS_PROPAGATION=rprivate', '-e', 'PACKTIVITY_AUTH_LOCATION=None', '-e', 'YADAGE_SCHEMA_LOAD_TOKEN=None', 'recast/recastatlas:v0.1.0', 'sh', '-c', 'mkdir -p ~/.docker\necho \'{}\' > ~/.docker/config.json \ncat << \'EOF\' | yadage-run -f - \n{"dataarg": "recast-641382ce", "dataopts": {}, "initdata": {"did": 450253, "xsec_in_pb": 114.117, "dxaod_files_mc16a": ["/eos/project/r/recast/atlas/ANA-HDBS-2018-41/mc/scalar/mc16_13TeV.450253.MadGraphHerwig7EvtGen_NNPDF23LO_X400tohh_bbbb.deriv.DAOD_EXOT8.e7352_s3126_r9364_p4089/*"], "dxaod_files_mc16d": ["/eos/project/r/recast/atlas/ANA-HDBS-2018-41/mc/scalar/mc16_13TeV.450253.MadGraphHerwig7EvtGen_NNPDF23LO_X400tohh_bbbb.deriv.DAOD_EXOT8.e7352_s3126_r10201_p4089/*"], "dxaod_files_mc16e": ["/eos/project/r/recast/atlas/ANA-HDBS-2018-41/mc/scalar/mc16_13TeV.450253.MadGraphHerwig7EvtGen_NNPDF23LO_X400tohh_bbbb.deriv.DAOD_EXOT8.e7352_s3126_r10724_p4089/*"], "data_dir": "/eos/project/r/recast/atlas/ANA-HDBS-2018-41/data/NNT_FEB21/NNT_FEB21"}, "workflow": "workflow.yml", "toplevel": "/home/beojan/DPhil/hh4b/Recast/recast-resonant/specs", "visualize": true}\nEOF\n']' returned non-zero exit status 1.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment