Skip to content

Instantly share code, notes, and snippets.

View matthewrmshin's full-sized avatar
🏠
😃

Matt Shin matthewrmshin

🏠
😃
18 followers · 46 following
  • Met Office
  • Exeter, UK
View GitHub Profile
@matthewrmshin
matthewrmshin / my-cprofile
Created January 28, 2021 15:11
Python cProfile script
#!/bin/bash
set -euo pipefail
finally() {
trap '' ERR
trap '' EXIT
if [[ -n "${WORKD:-}" ]]; then
rm -fr "${WORKD}"
fi
}
@matthewrmshin
matthewrmshin / install-condaenv
Last active May 4, 2020 11:04
A set of Python applications run by a Cylc suite using a shared environment in tasks. This script installs the environment in the suite.
#!/bin/bash
#------------------------------------------------------------------------------
# This script is used by a Python project that is also a Cylc suite.
# An initial task in the suite runs this script to build and install the Python
# project and its environment in a self-contained Conda environment under
# `~/cylc-run/SUITE/share/condaenv/`.
# Subsequent tasks activates this environnment using:
# set +u
# . "${USER_CONDA_PROFILE}"
# conda activate "${SUITE_CONDA_ENV}"
@matthewrmshin
matthewrmshin / datetime.sed
Last active January 6, 2020 14:54
Convert YYYYMMDDhhmm to YYYY-MM-DD hh:mm:00 in column 1
#!/usr/bin/sed -f
# To use:
# 1. Download or copy this script.
# 2. Save it in a suitable location. E.g. ./datetime.sed
# 3. Ensure this script is executable.
# 4. Run: ./datetime.sed IN-FILE > OUT-FILE
s/^\([0-9]\{4\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)/\1-\2-\3 \4:\5:00/
@matthewrmshin
matthewrmshin / wfs=remote-job=wfs
Created July 26, 2019 13:40
WFS=>Remote Job=>WFS sequence diagram
title WFS => Remote Job => WFS
participantgroup #lightblue **WFS Host**
participant WFS
end
participantgroup #lightgrey **Remote Job Host**
participant Job Submit
database File System
participant Batch System
participant Job
@matthewrmshin
matthewrmshin / triage-20190308.md
Last active March 8, 2019 15:20 — forked from kinow/triage-20190308.md
Cylc issues triage performed on 2019-03-08
@matthewrmshin
matthewrmshin / mysleepypool.py
Created August 21, 2018 11:52
Quick Python concurrent multiprocessing.Process example
#!/usr/bin/env python
from multiprocessing import Process
import sys
from time import sleep
POLL_DURATION = 0.01
def worker(sleeptime, message):
@matthewrmshin
matthewrmshin / mybunch.py
Last active May 25, 2018 09:25
How to manage a pool of subprocesses in Python
#!/usr/bin/python
import os
from pipes import quote
from subprocess import Popen, PIPE
import sys
from time import sleep
N_PROCS = 4 # adjust this or make this an option
@matthewrmshin
matthewrmshin / test-trim
Created December 4, 2013 21:27
Test correctness of a script trim function.
#!/usr/bin/python
import re
import unittest
def trim(scr_in):
n_ws = 0
scr_out = ""
pre = ""
for line in scr_in.splitlines():
@matthewrmshin
matthewrmshin / cylc-suite-3000
Last active December 27, 2015 18:29
A Cylc suite of 3000 tasks per cycle
#!jinja2
title=Suite-3000
description=A suite of 3000 tasks per cycle
[cylc]
UTC mode=True # Ignore DST
[scheduling]
initial cycle time=20130101T00Z
final cycle time=20130201T00Z
[[dependencies]]
[[[PT12H]]]
@matthewrmshin
matthewrmshin / my-text-busy-test-p
Created July 18, 2013 13:51
An attempt to repeat the `/bin/sh: FILE: /bin/bash: bad interpreter: Text file busy` problem. Write 10000 script files and executes them using a pool 10 processes.
#!/usr/bin/python
from glob import glob
from multiprocessing import Pool
import os
import stat
from subprocess import call
from uuid import uuid4
def main():