Skip to content

Instantly share code, notes, and snippets.

Avatar

Johannes Buchner JohannesBuchner

View GitHub Profile
@JohannesBuchner
JohannesBuchner / rsyncprogress.py
Last active Sep 1, 2020
Progress bar for rsync
View rsyncprogress.py
"""
Progress bar for rsync
========================
Shows file progress and total progress as a progress bar.
Usage
---------
Run rsync with -P and pipe into this program. Example::
View benchmark.sh
# columns: id1,id2,id3,id4,id5,id6,v1,v2,v3
f=G1_1e7_1e2_0_0.csv
awk="time mawk"
# groupby simple
$awk -F, 'NR>1 { a[$1] += $7 } END {for (i in a) print i, a[i]}' $f >/dev/null
$awk -F, 'NR>1 { a[$1,$2] += $7 } END { for (comb in a) { split(comb,sep,SUBSEP); print sep[1], sep[2], a[sep[1],sep[2]]; }}' $f >/dev/null
$awk -F, 'NR>1 { a[$3] += $7; n[$3]++; b[$3] += $9; } END {for (i in a) print i, a[i], b[i]/n[i];}' $f >/dev/null
$awk -F, 'NR>1 { a[$4] += $7; n[$4]++; b[$4] += $8; } END {for (i in a) print i, a[i]/n[i], b[i]/n[i];}' $f >/dev/null
@JohannesBuchner
JohannesBuchner / cachestan.py
Created May 18, 2020
Build and cache Stan models smartly (ignoring changes in comments and white spaces)
View cachestan.py
import re
import pystan
import hashlib
import pickle
import os
def build_model(code):
lines = code.split("\n")
lines = [re.sub('//.*$', '', line).strip() for line in lines]
lines = [line.replace(' ', ' ').replace(' ', ' ').replace(' ', ' ')
@JohannesBuchner
JohannesBuchner / run_processes_when_idle.sh
Created May 12, 2020
Stop/resume processes when the user is active/inactive in Linux/Xorg
View run_processes_when_idle.sh
#!/bin/bash
# how to identify the processes to STOP/CONT
# this is any part of process command line
PROCESS="myscript.py"
# run this in one terminal:
# it tries to resume the processes every so often
while sleep 10m; do
pgrep -f "${PROCESS}"|xargs -rt kill -CONT
@JohannesBuchner
JohannesBuchner / poissonerr.py
Last active Aug 12, 2019
Obtain correct error bars from observed counts (fraction in sample, detected events, etc.)
View poissonerr.py
"""
If you ever made a plot of "fractions" or "rates" with symmetric error
bars, like in the plot shown, this mini-tutorial is for you. Here is how
to compute correct error bars, so that uncertainties in the fractions
do not go below 0% or above 100%.
If you have a histogram for
instance, and you detected k objects in a given bin. What is the rate
underlying at quantiles q=0.1, 0.5(median), 0.9?
"""
@JohannesBuchner
JohannesBuchner / mcmc.py
Created Aug 5, 2019
Minimalistic MCMC implementation
View mcmc.py
import numpy as np
def mcmc(logfunction, x0, nsteps, sigma_p):
samples = np.empty((nsteps, len(x0)))
logL0 = logfunction(x0)
naccepts = 0
for i in range(nsteps):
x1 = np.random.normal(x0, sigma_p)
logL1 = logfunction(x1)
if logL1 - logL0 > np.log(np.random.uniform()):
@JohannesBuchner
JohannesBuchner / toh5.py
Created Jun 2, 2019
Translate a csv/npy/csv.gz integer file to efficiently compressed HDF5
View toh5.py
import sys
import numpy
import h5py
filename = sys.argv[1]
outfilename = filename.replace('.npy', '').replace('.gz', '').replace('.csv', '') + '.h5'
if filename.endswith('.npy'):
print('loading NPY...')
data = numpy.load(filename)
else:
View fetchswiftspectrum.py
"""
Download custom spectrum from http://www.swift.ac.uk/, programmatically
How to use:
$ python fetchswiftspectrum.py grbid timeslicespec
$ wget -nc --continue $(cat customspec.url)/a.tar.gz
"""
View makegaia.sh
wget -nc http://cdn.gea.esac.esa.int/Gaia/gdr2/gaia_source/csv/
grep -Eo GaiaSource_.*.csv.gz index.html | while read i; do
if [ -e $i ]; then
true
else
wget "http://cdn.gea.esac.esa.int/Gaia/gdr2/gaia_source/csv/$i" -O - | zcat | LC_ALL=C awk -F, '{print $2,$6,$8,sqrt($7**2+$9**2),$48}' | tail -n +2 | gzip > $i.tmp && mv $i.tmp $i
fi
@JohannesBuchner
JohannesBuchner / publish.sh
Created Jan 19, 2019
script for releasing packages on PyPI and publishing sphinx documentation on github pages
View publish.sh
#!/bin/bash
# Author: Johannes Buchner (C) 2013
# tool for publishing sphinx documentation on github
# and releasing packages on PyPI
case "$1" in
doc)
# see https://help.github.com/articles/creating-project-pages-manually
make -C doc/ html &&
git checkout gh-pages &&
You can’t perform that action at this time.