Skip to content

Instantly share code, notes, and snippets.

Avatar

Will Furnass willfurnass

View GitHub Profile
@willfurnass
willfurnass / python_debug_env.md
Last active Aug 25, 2020
Setting up a Python debug environment to investigate a segfaulting Python C extension
View python_debug_env.md

Notes on setting up a Python debug environment

Used to investigate a segfault in a Python C extension within a Python package.

  1. Install debug Python build using pyenv then check is available for use:

    pyenv install -g 3.8.5
    pyenv versions
    
@willfurnass
willfurnass / export_data.py
Created Feb 20, 2019
Python snippet to export data to a file in a particular format
View export_data.py
import os
from typing import Dict
import numpy as np
def export_data(data: Dict[np.ndarray],
filename: str,
cols_per_line: int = 8) -> None:
"""Export dictionary of 1d float arrays to a file.
@willfurnass
willfurnass / jsv_crlf_rejecter.tcl
Created Apr 9, 2018
Grid Engine JSV snippet to reject jobs containing CRLF
View jsv_crlf_rejecter.tcl
#!/usr/bin/tclsh
set sge_root $env(SGE_ROOT)
source "$sge_root/util/resources/jsv/jsv_include.tcl"
proc jsv_on_start {} {
jsv_send_env
}
proc lexists name {
@willfurnass
willfurnass / paramiko_test.py
Last active May 18, 2018
Submitting a job to a remote Grid Engine cluster using Paramiko (Python SSH client)
View paramiko_test.py
import tempfile
import os
import time
import paramiko
# Hostname of a 'submission host' in the Grid Engine cluster we want to submit a job to
hostname = 'sharc.shef.ac.uk'
# Username for logging in to the cluster via SSH
username = "te1st"
@willfurnass
willfurnass / dss2emails.sh
Last active Apr 10, 2021
Extract usernames and emails from a Synology DSS config dump file (.dss)
View dss2emails.sh
#!/bin/bash
# Will Furnass
# Oct 2017
if [[ $# -lt 1 ]]; then
echo 1>&2 "Extract list of user email addresses from Synology DSS '.dss' config dump"
exit 1
fi
dss_path="$1"
@willfurnass
willfurnass / README.md
Created Jun 1, 2017
Separating Reveal.js from your presentation content using git subrepo
View README.md

Initialise a git repository:

cd somedir
git init

Install git subrepo if you haven't done so already:

https://github.com/ingydotnet/git-subrepo#installation
@willfurnass
willfurnass / sge_cheatsheet.md
Last active Apr 7, 2021
(Son of) Grid Engine tips and tricks
View sge_cheatsheet.md

Random Grid Engine tips and tricks

The following work with Son of Grid Engine (SGE) 8.1.9 as configured on the University of Sheffield's ShARC and Iceberg clusters.

Jobs with dependancies

You can use the -hold_jid <<job-name or job-name>> option to make jobs run only when other jobs have finished, rather than having jobs start and sit waiting for other tasks to complete.

View tmux_cheatsheet.md

tmux tips

Default prefix: <ctrl>b

Session management

  • tmux new -s session_name - creates a new tmux session named session_name
  • tmux a - attach to first available session
  • tmux attach -t session_name - attaches to an existing tmux session named session_name
  • tmux switch -t session_name - switches to an existing session named session_name
@willfurnass
willfurnass / qrshx
Created Apr 19, 2017
Simple wrapper for Sun Grid Engine's qrsh that propagates SGE environment variables and enables X forwarding
View qrshx
#!/bin/sh
# David Jones and Will Furnass (University of Sheffield)
# `qrsh` is often the most convenient utility for starting interactive shell
# sessions on Sun/Son of Grid Engine (SGE) clusters but it has some limitations (as
# configured on the ShARC/Iceberg clusters):
#
# - none of the standard SGE environment vars (e.g. JOB_ID, PE_HOSTFILE,
# NSLOTS) that are defined in qsub/qsh batch/interactive sessions are set
# (annoying but not the end of the world);
@willfurnass
willfurnass / ruffus_simple_test.py
Created Jan 11, 2017
Simple test of the Ruffus workflow manager
View ruffus_simple_test.py
#!/usr/bin/env python
from __future__ import print_function
from ruffus import transform, suffix, pipeline_run
# A test of using Ruffus to locally run a very simple pipeline comprised of
# tasks defined as Python functions
# Can run this from a conda environnment created using
# conda create -n drmaatest -c bioconda python=2.7 ruffus drmaa