Skip to content

Instantly share code, notes, and snippets.

View matthewrmshin's full-sized avatar
🏠
😃

Matt Shin matthewrmshin

🏠
😃
18 followers · 46 following
  • Met Office
  • Exeter, UK
View GitHub Profile
@matthewrmshin
matthewrmshin / my-cprofile
Created January 28, 2021 15:11
Python cProfile script
#!/bin/bash
set -euo pipefail
finally() {
trap '' ERR
trap '' EXIT
if [[ -n "${WORKD:-}" ]]; then
rm -fr "${WORKD}"
fi
}
@matthewrmshin
matthewrmshin / install-condaenv
Last active May 4, 2020 11:04
A set of Python applications run by a Cylc suite using a shared environment in tasks. This script installs the environment in the suite.
#!/bin/bash
#------------------------------------------------------------------------------
# This script is used by a Python project that is also a Cylc suite.
# An initial task in the suite runs this script to build and install the Python
# project and its environment in a self-contained Conda environment under
# `~/cylc-run/SUITE/share/condaenv/`.
# Subsequent tasks activates this environnment using:
# set +u
# . "${USER_CONDA_PROFILE}"
# conda activate "${SUITE_CONDA_ENV}"
@matthewrmshin
matthewrmshin / datetime.sed
Last active January 6, 2020 14:54
Convert YYYYMMDDhhmm to YYYY-MM-DD hh:mm:00 in column 1
#!/usr/bin/sed -f
# To use:
# 1. Download or copy this script.
# 2. Save it in a suitable location. E.g. ./datetime.sed
# 3. Ensure this script is executable.
# 4. Run: ./datetime.sed IN-FILE > OUT-FILE
s/^\([0-9]\{4\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)/\1-\2-\3 \4:\5:00/
@matthewrmshin
matthewrmshin / wfs=remote-job=wfs
Created July 26, 2019 13:40
WFS=>Remote Job=>WFS sequence diagram
title WFS => Remote Job => WFS
participantgroup #lightblue **WFS Host**
participant WFS
end
participantgroup #lightgrey **Remote Job Host**
participant Job Submit
database File System
participant Batch System
participant Job
@matthewrmshin
matthewrmshin / triage-20190308.md
Last active March 8, 2019 15:20 — forked from kinow/triage-20190308.md
Cylc issues triage performed on 2019-03-08
@matthewrmshin
matthewrmshin / mysleepypool.py
Created August 21, 2018 11:52
Quick Python concurrent multiprocessing.Process example
#!/usr/bin/env python
from multiprocessing import Process
import sys
from time import sleep
POLL_DURATION = 0.01
def worker(sleeptime, message):
@matthewrmshin
matthewrmshin / mybunch.py
Last active May 25, 2018 09:25
How to manage a pool of subprocesses in Python
#!/usr/bin/python
import os
from pipes import quote
from subprocess import Popen, PIPE
import sys
from time import sleep
N_PROCS = 4 # adjust this or make this an option
@matthewrmshin
matthewrmshin / test-cylc-scan.sh
Created July 30, 2015 09:15
Test cylc scan
#!/bin/bash
set -eu
begin() {
for I in $(seq -w 1 30); do
mkdir -p "${HOME}/cylc-run/scan-test-${I}"
cat >"${HOME}/cylc-run/scan-test-${I}/suite.rc" <<'__SUITERC__'
[scheduling]
[[dependencies]]
graph = t1
@matthewrmshin
matthewrmshin / time the time
Created July 8, 2015 13:13
Proof that time.time is quicker than datetime.datetime.utcnow
#!/usr/bin/python
from datetime import datetime
from time import clock, time
N_TIMES = 100000
def main():
begin = clock()
@matthewrmshin
matthewrmshin / cylc-suite-100
Created July 18, 2014 14:34
A cylc suite of 100 tasks per cycle
#!jinja2
title=Basic suite
description=A basic suite of 100 tasks per cycle
[cylc]
UTC mode=True # Ignore DST
[scheduling]
initial cycle time=20130101
final cycle time=20130110
#runahead limit=12
[[dependencies]]