Skip to content

Instantly share code, notes, and snippets.

View willfurnass's full-sized avatar

Will Furnass willfurnass

View GitHub Profile
willfurnass /
Last active August 25, 2020 17:08
Setting up a Python debug environment to investigate a segfaulting Python C extension

Notes on setting up a Python debug environment

Used to investigate a segfault in a Python C extension within a Python package.

  1. Install debug Python build using pyenv then check is available for use:

    pyenv install -g 3.8.5
    pyenv versions
willfurnass /
Created February 20, 2019 18:00
Python snippet to export data to a file in a particular format
import os
from typing import Dict
import numpy as np
def export_data(data: Dict[np.ndarray],
filename: str,
cols_per_line: int = 8) -> None:
"""Export dictionary of 1d float arrays to a file.
willfurnass / jsv_crlf_rejecter.tcl
Created April 9, 2018 12:46
Grid Engine JSV snippet to reject jobs containing CRLF
set sge_root $env(SGE_ROOT)
source "$sge_root/util/resources/jsv/jsv_include.tcl"
proc jsv_on_start {} {
proc lexists name {
willfurnass /
Last active February 11, 2022 11:12
Submitting a job to a remote Grid Engine cluster using Paramiko (Python SSH client)
import tempfile
import os
import time
import paramiko
# Hostname of a 'submission host' in the Grid Engine cluster we want to submit a job to
hostname = ''
# Username for logging in to the cluster via SSH
username = "te1st"
willfurnass /
Last active February 20, 2022 12:03
Extract usernames and emails from a Synology DSS config dump file (.dss)
# Will Furnass
# Oct 2017
if [[ $# -lt 1 ]]; then
echo 1>&2 "Extract list of user email addresses from Synology DSS '.dss' config dump"
exit 1
willfurnass /
Created June 1, 2017 11:21
Separating Reveal.js from your presentation content using git subrepo

Initialise a git repository:

cd somedir
git init

Install git subrepo if you haven't done so already:
willfurnass /
Last active April 19, 2023 15:00
(Son of) Grid Engine tips and tricks

Random Grid Engine tips and tricks

The following work with Son of Grid Engine (SGE) 8.1.9 as configured on the University of Sheffield's ShARC and Iceberg clusters.

Jobs with dependancies

You can use the -hold_jid <<job-name or job-name>> option to make jobs run only when other jobs have finished, rather than having jobs start and sit waiting for other tasks to complete.

tmux tips

Default prefix: <ctrl>b

Session management

  • tmux new -s session_name - creates a new tmux session named session_name
  • tmux a - attach to first available session
  • tmux attach -t session_name - attaches to an existing tmux session named session_name
  • tmux switch -t session_name - switches to an existing session named session_name
willfurnass / qrshx
Created April 19, 2017 14:44
Simple wrapper for Sun Grid Engine's qrsh that propagates SGE environment variables and enables X forwarding
# David Jones and Will Furnass (University of Sheffield)
# `qrsh` is often the most convenient utility for starting interactive shell
# sessions on Sun/Son of Grid Engine (SGE) clusters but it has some limitations (as
# configured on the ShARC/Iceberg clusters):
# - none of the standard SGE environment vars (e.g. JOB_ID, PE_HOSTFILE,
# NSLOTS) that are defined in qsub/qsh batch/interactive sessions are set
# (annoying but not the end of the world);
willfurnass /
Created January 11, 2017 12:27
Simple test of the Ruffus workflow manager
#!/usr/bin/env python
from __future__ import print_function
from ruffus import transform, suffix, pipeline_run
# A test of using Ruffus to locally run a very simple pipeline comprised of
# tasks defined as Python functions
# Can run this from a conda environnment created using
# conda create -n drmaatest -c bioconda python=2.7 ruffus drmaa