Skip to content

Instantly share code, notes, and snippets.

View natefoo's full-sized avatar

Nate Coraor natefoo

69 followers · 1 following
View GitHub Profile
@natefoo
natefoo / run_tag_test_import.sh-session
Last active January 15, 2021 19:51
(ansible)natefoo@fanboy% ansible-playbook -i localhost, tag_test_import.yml
PLAY [localhost] ************************************************************************************************************************************************************************************************************************************
TASK [Gathering Facts] ******************************************************************************************************************************************************************************************************************************
[WARNING]: Platform darwin on host localhost is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change the meaning of that path. See
https://docs.ansible.com/ansible/2.10/reference_appendices/interpreter_discovery.html for more information.
ok: [localhost]
TASK [debug] ****************************************************************************************************************
@natefoo
natefoo / twobit.log
Created December 14, 2020 22:44
[g2main@galaxy-web-05 ~]$ grep twobit /srv/galaxy/main/log/zergling0.log
galaxy.tools.data DEBUG 2020-12-09 19:57:30,617 [p:29736,w:0,m:0] [MainThread] Loaded 19 lines from '/cvmfs/data.galaxyproject.org/byhand/location/twobit.loc' for 'twobit'
galaxy.tools.data DEBUG 2020-12-09 19:57:30,617 [p:29736,w:0,m:0] [MainThread] Loaded tool data table 'twobit' from file '/srv/galaxy/main/config/tool_data_table_conf.xml'
galaxy.tools.data DEBUG 2020-12-09 19:57:30,713 [p:29736,w:0,m:0] [MainThread] Loaded 21 lines from '/cvmfs/data.galaxyproject.org/managed/location/twobit.loc' for 'twobit'
galaxy.tools.data DEBUG 2020-12-09 19:57:30,713 [p:29736,w:0,m:0] [MainThread] Loading another instance of data table 'twobit' from file '/cvmfs/data.galaxyproject.org/managed/location/tool_data_table_conf.xml', attempting to merge content.
galaxy.tools.data DEBUG 2020-12-09 19:57:31,891 [p:29736,w:0,m:0] [MainThread] Loaded 0 lines from '/galaxy-repl/main/tool_data/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_twobit_builder/
@natefoo
natefoo / crash.md
Created November 17, 2020 18:33
Grafana image renderer timeout

Turns out it's probably Chromium crashing, as determined from running the command line (obtained from ps(1) after clicking to generate an image) and running it by hand:

-bash-4.2$ /var/lib/grafana/plugins/grafana-image-renderer/chrome-linux/chrome --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=TranslateUI,BlinkGenPropertyTrees --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --disable-sync --force-color-profile=srgb --metrics-recording-only --no-first-run --enable-automation --password-store=basic --use-mock-keychain --headless --hide-scrollbars --mute-audio about:blank --no-sandbox --remote-debu
@natefoo
natefoo / Multi-host Galaxy Server.md
Last active November 9, 2020 20:37

Multi-host Galaxy Server

usegalaxy.org uses a five host setup:

  • 2 x hosts for serving web requests
  • 2 x hosts for handling jobs and Pulsar staging
  • 1 x host for the database

Web requests are balanced only with a DNS round-robin (i.e. there are two A records for usegalaxy.org), which is not ideal since it does not do any load balancing or health checks. A better setup would include a 6th host with nginx proxying the web hosts.

@natefoo
natefoo / pulsar-deps.pytb
Created September 22, 2020 17:59
2020-09-22 17:50:57,516 ERROR [pulsar.managers.stateful][[manager=jetstream_tacc]-[action=preprocess]-[job=1306569]] Failed job preprocessing for job 1306569:
Traceback (most recent call last):
File "/srv/pulsar/test/venv/lib64/python3.6/site-packages/pulsar/managers/stateful.py", line 132, in _handling_of_preprocessing_state
**launch_kwds
File "/srv/pulsar/test/venv/lib64/python3.6/site-packages/pulsar/managers/queued_drmaa.py", line 21, in launch
setup_params=setup_params,
File "/srv/pulsar/test/venv/lib64/python3.6/site-packages/pulsar/managers/base/base_drmaa.py", line 65, in _build_template_attributes
setup_params=setup_params
File "/srv/pulsar/test/venv/lib64/python3.6/site-packages/pulsar/managers/base/directory.py", line 120, in _setup_job_file
command_line = self._expand_command_line(command_line, dependencies_description, job_directory=self.job_directory(job_id).job_directory)
@natefoo
natefoo / gist:d06dbcfce8593f52a10adf56a9c77f0e
Created July 7, 2020 18:21
https://zenodo.org/record/3928735/files/AM1.fastq?download=1
https://zenodo.org/record/3928735/files/AM2.fastq?download=1
https://zenodo.org/record/3928735/files/AM3.fastq?download=1
https://zenodo.org/record/3928735/files/EM1.fastq?download=1
https://zenodo.org/record/3928735/files/EM3.fastq?download=1
https://zenodo.org/record/3928735/files/EM4.fastq?download=1
https://zenodo.org/record/3928735/files/EM5.fastq?download=1
@natefoo
natefoo / requeue-job.py
Created April 15, 2020 14:27
if options.requeue_job:
from galaxy.util import directory_hash_id
job_id = options.requeue_job
job = model.context.current.query(model.Job).enable_eagerloads(False).get(job_id)
job_dir_hash = os.path.join(*directory_hash_id(job_id))
job_working_dir = os.path.join(JOB_WORKING_DIR, job_dir_hash, str(job_id))
print('Attempting to requeue job %s/%s' % (job_id, job.job_runner_external_id))
if os.path.exists(job_working_dir):
cleared_dir = os.path.join(JOB_WORKING_DIR, '_cleared_contents', job_dir_hash, str(job_id))
if not os.path.exists(cleared_dir):
@natefoo
natefoo / directory.py.diff
Created April 14, 2020 18:34
CVMFS Parrot Pulsar hack
diff --git a/pulsar/managers/base/directory.py b/pulsar/managers/base/directory.py
index 4a463dc..e527501 100644
--- a/pulsar/managers/base/directory.py
+++ b/pulsar/managers/base/directory.py
@@ -19,6 +19,14 @@ JOB_FILE_TOOL_ID = "tool_id"
JOB_FILE_TOOL_VERSION = "tool_version"
JOB_FILE_CANCELLED = "cancelled"
JOB_FILE_COMMAND_LINE = "command_line"
+JOB_WRAPPER_TEMPLATE = """#!/bin/sh
+PARROT_CVMFS_REPO="data.galaxyproject.org:url=http://cvmfs1-tacc0.galaxyproject.org/cvmfs/data.galaxyproject.org/,pubkey=$HOME/data.pub \
@natefoo
natefoo / main-slots-tools.txt
Created April 9, 2020 17:48
abyss-pe
align_families
alleyoop
bg_diamond
bg_diamond_makedb
bio_hansel
bowtie2
bowtie_color_wrapper
bowtie_wrapper
busco
@natefoo
natefoo / map_resources.py
Created March 4, 2020 09:34
import logging
from galaxy.jobs.mapper import JobMappingException
log = logging.getLogger(__name__)
DESTINATION_IDS = {
1 : 'slurm',
2 : 'slurm-2c'
}
FAILURE_MESSAGE = 'This tool could not be run because of a misconfiguration in the Galaxy job running system, please report this error'