Skip to content

Instantly share code, notes, and snippets.

View willfurnass's full-sized avatar

Will Furnass willfurnass

View GitHub Profile
willfurnass /
Last active April 19, 2023 15:00
(Son of) Grid Engine tips and tricks
willfurnass / qrshx
Created April 19, 2017 14:44
Simple wrapper for Sun Grid Engine's qrsh that propagates SGE environment variables and enables X forwarding
View qrshx
# David Jones and Will Furnass (University of Sheffield)
# `qrsh` is often the most convenient utility for starting interactive shell
# sessions on Sun/Son of Grid Engine (SGE) clusters but it has some limitations (as
# configured on the ShARC/Iceberg clusters):
# - none of the standard SGE environment vars (e.g. JOB_ID, PE_HOSTFILE,
# NSLOTS) that are defined in qsub/qsh batch/interactive sessions are set
# (annoying but not the end of the world);
willfurnass /
Last active February 20, 2022 12:03
Extract usernames and emails from a Synology DSS config dump file (.dss)
# Will Furnass
# Oct 2017
if [[ $# -lt 1 ]]; then
echo 1>&2 "Extract list of user email addresses from Synology DSS '.dss' config dump"
exit 1
willfurnass /
Last active February 11, 2022 11:12
Submitting a job to a remote Grid Engine cluster using Paramiko (Python SSH client)
import tempfile
import os
import time
import paramiko
# Hostname of a 'submission host' in the Grid Engine cluster we want to submit a job to
hostname = ''
# Username for logging in to the cluster via SSH
username = "te1st"
willfurnass /
Last active August 25, 2020 17:08
Setting up a Python debug environment to investigate a segfaulting Python C extension

Notes on setting up a Python debug environment

Used to investigate a segfault in a Python C extension within a Python package.

  1. Install debug Python build using pyenv then check is available for use:

    pyenv install -g 3.8.5
    pyenv versions
willfurnass /
Created February 20, 2019 18:00
Python snippet to export data to a file in a particular format
import os
from typing import Dict
import numpy as np
def export_data(data: Dict[np.ndarray],
filename: str,
cols_per_line: int = 8) -> None:
"""Export dictionary of 1d float arrays to a file.
willfurnass / jsv_crlf_rejecter.tcl
Created April 9, 2018 12:46
Grid Engine JSV snippet to reject jobs containing CRLF
View jsv_crlf_rejecter.tcl
set sge_root $env(SGE_ROOT)
source "$sge_root/util/resources/jsv/jsv_include.tcl"
proc jsv_on_start {} {
proc lexists name {
willfurnass /
Created June 1, 2017 11:21
Separating Reveal.js from your presentation content using git subrepo

Initialise a git repository:

cd somedir
git init

Install git subrepo if you haven't done so already:

tmux tips

Default prefix: <ctrl>b

Session management

  • tmux new -s session_name - creates a new tmux session named session_name
  • tmux a - attach to first available session
  • tmux attach -t session_name - attaches to an existing tmux session named session_name
  • tmux switch -t session_name - switches to an existing session named session_name
willfurnass /
Last active January 11, 2017 13:40
Ruffus + DRMAA test
#!/usr/bin/env python
# A test of whether Ruffus can be used to submit jobs to a cluster using the
# DRMAA API, sharing the DRMAA session between threads to allow multiple jobs to
# be submitted at once without burdening the head node with multiple DRMAA
# sessions.
# Can run this from a conda environnment created using
# conda create -n drmaatest -c bioconda python=2.7 ruffus drmaa