Skip to content

Instantly share code, notes, and snippets.

Avatar

Johannes Buchner JohannesBuchner

View GitHub Profile
@JohannesBuchner
JohannesBuchner / cmdcache.py
Created Mar 1, 2021
Cache/Memoize any command line program. Keeps stdout, stderr and exit code, env aware.
View cmdcache.py
import sys, os
import joblib
import subprocess
mem = joblib.Memory('.', verbose=False)
@mem.cache
def run_cmd(args, env):
process = subprocess.run(args, capture_output=True, text=True)
return process.stdout, process.stderr, process.returncode
@JohannesBuchner
JohannesBuchner / joss_make_latex.sh
Created Jan 23, 2021
Make LaTeX and PDF from JOSS markdown papers
View joss_make_latex.sh
#!/bin/bash
# you need to install:
# pip install openbases
# sudo apt install texlive-xetex pandoc pandoc-citeproc
PDF_INFILE=paper.md
PDF_LOGO=logo.png
PDF_OUTFILE=paper.pdf
TEX_OUTFILE=paper.tex
@JohannesBuchner
JohannesBuchner / xray_opt_gif.sh
Created Dec 18, 2020
Make a gif flipping between an X-ray and optical image at some coordinate
View xray_opt_gif.sh
#!/bin/bash
# example usage:
# bash xray_opt_gif.sh 155.87737 +19.86508
RA=$1
DEC=$2
wget -nc "https://alasky.unistra.fr/hips-thumbnails/thumbnail?ra=${RA}&dec=${DEC}&fov=0.21750486127986932&width=500&height=500&hips_kw=CDS%2FP%2FSDSS9%2Fcolor" -O opt.jpg
wget -nc "https://alasky.unistra.fr/hips-thumbnails/thumbnail?ra=${RA}&dec=${DEC}&fov=0.21750486127986932&width=500&height=500&hips_kw=xcatdb%2FP%2FXMM%2FPN%2Fcolor" -O xmm.jpg
View benchmark.sh
# columns: id1,id2,id3,id4,id5,id6,v1,v2,v3
f=G1_1e7_1e2_0_0.csv
awk="time mawk"
# groupby simple
$awk -F, 'NR>1 { a[$1] += $7 } END {for (i in a) print i, a[i]}' $f >/dev/null
$awk -F, 'NR>1 { a[$1,$2] += $7 } END { for (comb in a) { split(comb,sep,SUBSEP); print sep[1], sep[2], a[sep[1],sep[2]]; }}' $f >/dev/null
$awk -F, 'NR>1 { a[$3] += $7; n[$3]++; b[$3] += $9; } END {for (i in a) print i, a[i], b[i]/n[i];}' $f >/dev/null
$awk -F, 'NR>1 { a[$4] += $7; n[$4]++; b[$4] += $8; } END {for (i in a) print i, a[i]/n[i], b[i]/n[i];}' $f >/dev/null
@JohannesBuchner
JohannesBuchner / cachestan.py
Created May 18, 2020
Build and cache Stan models smartly (ignoring changes in comments and white spaces)
View cachestan.py
import re
import pystan
import hashlib
import pickle
import os
def build_model(code):
lines = code.split("\n")
lines = [re.sub('//.*$', '', line).strip() for line in lines]
lines = [line.replace(' ', ' ').replace(' ', ' ').replace(' ', ' ')
@JohannesBuchner
JohannesBuchner / run_processes_when_idle.sh
Created May 12, 2020
Stop/resume processes when the user is active/inactive in Linux/Xorg
View run_processes_when_idle.sh
#!/bin/bash
# how to identify the processes to STOP/CONT
# this is any part of process command line
PROCESS="myscript.py"
# run this in one terminal:
# it tries to resume the processes every so often
while sleep 10m; do
pgrep -f "${PROCESS}"|xargs -rt kill -CONT
@JohannesBuchner
JohannesBuchner / poissonerr.py
Last active Aug 12, 2019
Obtain correct error bars from observed counts (fraction in sample, detected events, etc.)
View poissonerr.py
"""
If you ever made a plot of "fractions" or "rates" with symmetric error
bars, like in the plot shown, this mini-tutorial is for you. Here is how
to compute correct error bars, so that uncertainties in the fractions
do not go below 0% or above 100%.
If you have a histogram for
instance, and you detected k objects in a given bin. What is the rate
underlying at quantiles q=0.1, 0.5(median), 0.9?
"""
@JohannesBuchner
JohannesBuchner / mcmc.py
Created Aug 5, 2019
Minimalistic MCMC implementation
View mcmc.py
import numpy as np
def mcmc(logfunction, x0, nsteps, sigma_p):
samples = np.empty((nsteps, len(x0)))
logL0 = logfunction(x0)
naccepts = 0
for i in range(nsteps):
x1 = np.random.normal(x0, sigma_p)
logL1 = logfunction(x1)
if logL1 - logL0 > np.log(np.random.uniform()):
@JohannesBuchner
JohannesBuchner / toh5.py
Created Jun 2, 2019
Translate a csv/npy/csv.gz integer file to efficiently compressed HDF5
View toh5.py
import sys
import numpy
import h5py
filename = sys.argv[1]
outfilename = filename.replace('.npy', '').replace('.gz', '').replace('.csv', '') + '.h5'
if filename.endswith('.npy'):
print('loading NPY...')
data = numpy.load(filename)
else:
View fetchswiftspectrum.py
"""
Download custom spectrum from http://www.swift.ac.uk/, programmatically
How to use:
$ python fetchswiftspectrum.py grbid timeslicespec
$ wget -nc --continue $(cat customspec.url)/a.tar.gz
"""