Skip to content

Instantly share code, notes, and snippets.

@JohannesBuchner
JohannesBuchner / joss_make_latex.sh
Created January 23, 2021 22:30
Make LaTeX and PDF from JOSS markdown papers
#!/bin/bash
# you need to install:
# pip install openbases
# sudo apt install texlive-xetex pandoc pandoc-citeproc
PDF_INFILE=paper.md
PDF_LOGO=logo.png
PDF_OUTFILE=paper.pdf
TEX_OUTFILE=paper.tex
@JohannesBuchner
JohannesBuchner / benchmark.sh
Last active January 1, 2021 07:54
awk solutions for simple groupby in https://h2oai.github.io/db-benchmark/
# columns: id1,id2,id3,id4,id5,id6,v1,v2,v3
f=G1_1e7_1e2_0_0.csv
awk="time mawk"
# groupby simple
$awk -F, 'NR>1 { a[$1] += $7 } END {for (i in a) print i, a[i]}' $f >/dev/null
$awk -F, 'NR>1 { a[$1,$2] += $7 } END { for (comb in a) { split(comb,sep,SUBSEP); print sep[1], sep[2], a[sep[1],sep[2]]; }}' $f >/dev/null
$awk -F, 'NR>1 { a[$3] += $7; n[$3]++; b[$3] += $9; } END {for (i in a) print i, a[i], b[i]/n[i];}' $f >/dev/null
$awk -F, 'NR>1 { a[$4] += $7; n[$4]++; b[$4] += $8; } END {for (i in a) print i, a[i]/n[i], b[i]/n[i];}' $f >/dev/null
@JohannesBuchner
JohannesBuchner / xray_opt_gif.sh
Created December 18, 2020 12:14
Make a gif flipping between an X-ray and optical image at some coordinate
#!/bin/bash
# example usage:
# bash xray_opt_gif.sh 155.87737 +19.86508
RA=$1
DEC=$2
wget -nc "https://alasky.unistra.fr/hips-thumbnails/thumbnail?ra=${RA}&dec=${DEC}&fov=0.21750486127986932&width=500&height=500&hips_kw=CDS%2FP%2FSDSS9%2Fcolor" -O opt.jpg
wget -nc "https://alasky.unistra.fr/hips-thumbnails/thumbnail?ra=${RA}&dec=${DEC}&fov=0.21750486127986932&width=500&height=500&hips_kw=xcatdb%2FP%2FXMM%2FPN%2Fcolor" -O xmm.jpg
@JohannesBuchner
JohannesBuchner / cachestan.py
Created May 18, 2020 11:32
Build and cache Stan models smartly (ignoring changes in comments and white spaces)
import re
import pystan
import hashlib
import pickle
import os
def build_model(code):
lines = code.split("\n")
lines = [re.sub('//.*$', '', line).strip() for line in lines]
lines = [line.replace(' ', ' ').replace(' ', ' ').replace(' ', ' ')
@JohannesBuchner
JohannesBuchner / run_processes_when_idle.sh
Created May 12, 2020 08:56
Stop/resume processes when the user is active/inactive in Linux/Xorg
#!/bin/bash
# how to identify the processes to STOP/CONT
# this is any part of process command line
PROCESS="myscript.py"
# run this in one terminal:
# it tries to resume the processes every so often
while sleep 10m; do
pgrep -f "${PROCESS}"|xargs -rt kill -CONT
@JohannesBuchner
JohannesBuchner / mcmc.py
Created August 5, 2019 09:58
Minimalistic MCMC implementation
import numpy as np
def mcmc(logfunction, x0, nsteps, sigma_p):
samples = np.empty((nsteps, len(x0)))
logL0 = logfunction(x0)
naccepts = 0
for i in range(nsteps):
x1 = np.random.normal(x0, sigma_p)
logL1 = logfunction(x1)
if logL1 - logL0 > np.log(np.random.uniform()):
@JohannesBuchner
JohannesBuchner / toh5.py
Created June 2, 2019 04:40
Translate a csv/npy/csv.gz integer file to efficiently compressed HDF5
import sys
import numpy
import h5py
filename = sys.argv[1]
outfilename = filename.replace('.npy', '').replace('.gz', '').replace('.csv', '') + '.h5'
if filename.endswith('.npy'):
print('loading NPY...')
data = numpy.load(filename)
else:
@JohannesBuchner
JohannesBuchner / fetchswiftspectrum.py
Last active March 15, 2019 12:21
Download Swift spectra
"""
Download custom spectrum from http://www.swift.ac.uk/, programmatically
How to use:
$ python fetchswiftspectrum.py grbid timeslicespec
$ wget -nc --continue $(cat customspec.url)/a.tar.gz
"""
@JohannesBuchner
JohannesBuchner / makegaia.sh
Created February 27, 2019 09:57
Download GAIA data
wget -nc http://cdn.gea.esac.esa.int/Gaia/gdr2/gaia_source/csv/
grep -Eo GaiaSource_.*.csv.gz index.html | while read i; do
if [ -e $i ]; then
true
else
wget "http://cdn.gea.esac.esa.int/Gaia/gdr2/gaia_source/csv/$i" -O - | zcat | LC_ALL=C awk -F, '{print $2,$6,$8,sqrt($7**2+$9**2),$48}' | tail -n +2 | gzip > $i.tmp && mv $i.tmp $i
fi
@JohannesBuchner
JohannesBuchner / publish.sh
Created January 19, 2019 16:23
script for releasing packages on PyPI and publishing sphinx documentation on github pages
#!/bin/bash
# Author: Johannes Buchner (C) 2013
# tool for publishing sphinx documentation on github
# and releasing packages on PyPI
case "$1" in
doc)
# see https://help.github.com/articles/creating-project-pages-manually
make -C doc/ html &&
git checkout gh-pages &&