Skip to content

Instantly share code, notes, and snippets.

View iamh2o's full-sized avatar

(major) john (major) iamh2o

View GitHub Profile
@nathanhaigh
nathanhaigh / interleave_fastq.sh
Last active March 28, 2022 09:11
Interleave reads from 2 FASTQ files and output to STDOUT.
#!/bin/bash
# Usage: interleave_fastq.sh f.fastq r.fastq > interleaved.fastq
#
# Interleaves the reads of two FASTQ files specified on the
# command line and outputs a single FASTQ file of STDOUT.
#
# Can interleave 100 million paired reads (200 million total
# reads; a 2 x 22Gbyte files), in memory (/dev/shm), in 6m54s (414s)
#
# Latest code: https://gist.github.com/4544979
@satoruhiga
satoruhiga / mov2gif.py
Last active August 24, 2022 08:43
convert mov to gif animation
#!/usr/bin/env python
'''
DEPENDENCIES:
$ brew install ffmpeg
$ brew install imagemagick
$ python ./mov2gif.py input.mov output.gif 15
'''
@knmkr
knmkr / search_snpedia.py
Created January 22, 2014 11:40
Example for calling SNPedia API from python
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import argparse
from wikitools import wiki # https://github.com/alexz-enwp/wikitools
from wikitools import page
def search_snpedia(snp):
"""
http://snpedia.com/index.php/Bulk
@bradrydzewski
bradrydzewski / generate_docker_cert.sh
Last active May 27, 2024 15:59
Generate trusted CA certificates for running Docker with HTTPS
#!/bin/bash
#
# Generates client and server certificates used to enable HTTPS
# remote authentication to a Docker daemon.
#
# See http://docs.docker.com/articles/https/
#
# To start the Docker Daemon:
#
# sudo docker -d \
@surjikal
surjikal / macos-openssh-portable.sh
Last active June 14, 2024 19:07
Compile OpenSSH Portable on OSX / MacOS
#!/usr/bin/env bash
brew install openssl
git clone git://anongit.mindrot.org/openssh.git
cd openssh
# The './configure' script does not exist, so we have to build it
autoreconf
@SamStudio8
SamStudio8 / acct_parse.py
Last active November 28, 2023 16:12
SGE QACCT Parsing
"""
Provides functions for the purpose of parsing a Sun Grid Engine (SGE)
accounting file for job metadata.
"""
__author__ = "Sam Nicholls <msn@aber.ac.uk>"
__copyright__ = "Copyright (c) Sam Nicholls"
__version__ = "0.0.32"
__maintainer__ = "Sam Nicholls <msn@aber.ac.uk>"
@Brainiarc7
Brainiarc7 / transient-clustering-gnu-parallel-sshfs.md
Last active April 21, 2024 05:16
How to set up a transient cluster using GNU parallel and SSHFS for distributed jobs (such as FFmpeg media encodes)

Transient compute clustering with GNU Parallel and sshfs:

GNU Parallel is a multipurpose program for running shell commands in parallel, which can often be used to replace shell script loops,find -exec, and find | xargs. It provides the --sshlogin and --sshloginfile options to farm out jobs to multiple hosts, as well as options for sending and retrieving static resources and and per-job input and output files.

For any particular task, however, keeping track of which files need to pushed to and retrieved from the remote hosts is somewhat of a hassle. Furthermore, cancelled or failed runs can leave garbage on the remote hosts, and if input and output files are large, sending them to local disk on the remote hosts is somewhat inefficient.

In a traditional cluster, this problem would be solved by giving all nodes access to a shared filesystem, usually with NFS or something more exotic. However, NFS doesn't wo

@fsteffenhagen
fsteffenhagen / sum_filesize.sh
Last active April 15, 2024 10:04
sum human readable file sizes with numfmt and awk
# Input: list of rows with format: "<filesize> filename", e.g.
# filesizes.txt
#######################
# 1000K file1.txt
# 200M file2.txt
# 2G file3.txt
#
# Output:
cat filesizes.txt | numfmt --from=iec | awk 'BEGIN {sum=0} {sum=sum+$1} END {printf "%.0f\n", sum}'
@adam-garcia
adam-garcia / README.md
Last active April 23, 2024 09:41
Viridis Color Palette Generator

Viridis Color Palette Generator

A simple tool for extracting discrete color palette values from the viridis collection of color palettes.

#!/bin/bash
set -ex
gcc_ver=$(gcc --version | head -n 1 | awk '{print $3}')
# Install gcc 4.9 devtoolset
if [[ ! "$gcc_ver" =~ 4\.9\.* ]]; then
sudo yum install epel-release -y
sudo yum install centos-release-scl -y