Skip to content

Instantly share code, notes, and snippets.

Avatar

Will Furnass willfurnass

View GitHub Profile
@willfurnass
willfurnass / sge_cheatsheet.md
Last active Aug 26, 2022
(Son of) Grid Engine tips and tricks
View sge_cheatsheet.md

Random Grid Engine tips and tricks

The following work with Son of Grid Engine (SGE) 8.1.9 as configured on the University of Sheffield's ShARC and Iceberg clusters.

Jobs with dependancies

You can use the -hold_jid <<job-name or job-name>> option to make jobs run only when other jobs have finished, rather than having jobs start and sit waiting for other tasks to complete.

@willfurnass
willfurnass / qrshx
Created Apr 19, 2017
Simple wrapper for Sun Grid Engine's qrsh that propagates SGE environment variables and enables X forwarding
View qrshx
#!/bin/sh
# David Jones and Will Furnass (University of Sheffield)
# `qrsh` is often the most convenient utility for starting interactive shell
# sessions on Sun/Son of Grid Engine (SGE) clusters but it has some limitations (as
# configured on the ShARC/Iceberg clusters):
#
# - none of the standard SGE environment vars (e.g. JOB_ID, PE_HOSTFILE,
# NSLOTS) that are defined in qsub/qsh batch/interactive sessions are set
# (annoying but not the end of the world);
@willfurnass
willfurnass / dss2emails.sh
Last active Feb 20, 2022
Extract usernames and emails from a Synology DSS config dump file (.dss)
View dss2emails.sh
#!/bin/bash
# Will Furnass
# Oct 2017
if [[ $# -lt 1 ]]; then
echo 1>&2 "Extract list of user email addresses from Synology DSS '.dss' config dump"
exit 1
fi
dss_path="$1"
@willfurnass
willfurnass / paramiko_test.py
Last active Feb 11, 2022
Submitting a job to a remote Grid Engine cluster using Paramiko (Python SSH client)
View paramiko_test.py
import tempfile
import os
import time
import paramiko
# Hostname of a 'submission host' in the Grid Engine cluster we want to submit a job to
hostname = 'sharc.shef.ac.uk'
# Username for logging in to the cluster via SSH
username = "te1st"
@willfurnass
willfurnass / python_debug_env.md
Last active Aug 25, 2020
Setting up a Python debug environment to investigate a segfaulting Python C extension
View python_debug_env.md

Notes on setting up a Python debug environment

Used to investigate a segfault in a Python C extension within a Python package.

  1. Install debug Python build using pyenv then check is available for use:

    pyenv install -g 3.8.5
    pyenv versions
    
@willfurnass
willfurnass / export_data.py
Created Feb 20, 2019
Python snippet to export data to a file in a particular format
View export_data.py
import os
from typing import Dict
import numpy as np
def export_data(data: Dict[np.ndarray],
filename: str,
cols_per_line: int = 8) -> None:
"""Export dictionary of 1d float arrays to a file.
@willfurnass
willfurnass / jsv_crlf_rejecter.tcl
Created Apr 9, 2018
Grid Engine JSV snippet to reject jobs containing CRLF
View jsv_crlf_rejecter.tcl
#!/usr/bin/tclsh
set sge_root $env(SGE_ROOT)
source "$sge_root/util/resources/jsv/jsv_include.tcl"
proc jsv_on_start {} {
jsv_send_env
}
proc lexists name {
@willfurnass
willfurnass / README.md
Created Jun 1, 2017
Separating Reveal.js from your presentation content using git subrepo
View README.md

Initialise a git repository:

cd somedir
git init

Install git subrepo if you haven't done so already:

https://github.com/ingydotnet/git-subrepo#installation
View tmux_cheatsheet.md

tmux tips

Default prefix: <ctrl>b

Session management

  • tmux new -s session_name - creates a new tmux session named session_name
  • tmux a - attach to first available session
  • tmux attach -t session_name - attaches to an existing tmux session named session_name
  • tmux switch -t session_name - switches to an existing session named session_name
View ruffus_simple_drmaa_test.py
#!/usr/bin/env python
# A test of whether Ruffus can be used to submit jobs to a cluster using the
# DRMAA API, sharing the DRMAA session between threads to allow multiple jobs to
# be submitted at once without burdening the head node with multiple DRMAA
# sessions.
# Can run this from a conda environnment created using
# conda create -n drmaatest -c bioconda python=2.7 ruffus drmaa