Skip to content

Instantly share code, notes, and snippets.

@rossjones
rossjones / purge-ipfs-pins.sh
Created August 22, 2023 13:22
Deletes old bacalhau job output
# Finds all of the job output that is older than 1month
# old and unpins it from IPFS so that the GC can clean
# it up.
#
# Requires JQ (https://jqlang.github.io/jq/download/)
# Get the unix timestamp (since the epoch) for the date 1 month ago.
LAST_MONTH=$(date -v"-1m" +%s)
# List all of the jobs with the newest first, and retrieve up to 1000 jobs
@rossjones
rossjones / wasmlogs.md
Created March 22, 2023 11:59
Streaming wasm logs

Currently only docker logs are streamable to the command line, and we wish to support this for WASM jobs as well.

Docker achieves these streaming logs by allowing the execution process to write to a ring buffer which will write to a log-driver based on configuration. By default, this driver is a JSON file and so docker will write log entries in the following format as jsonl:

{
"log": "\u001b[32mThe Project Gutenberg EBook of The Cosmic Computer, by Henry Beam Piper\n",
"stream": "stdout",
"time": "2023-03-17T11:37:08.764363458Z"
set -sg escape-time 0

Keybase proof

I hereby claim:

  • I am rossjones on github.
  • I am rossjones (https://keybase.io/rossjones) on keybase.
  • I have a public key whose fingerprint is 8127 12C1 15D7 87A5 F4D0 B83D 11D2 232C CA67 1ABC

To claim this, I am signing this object:

Keybase proof

I hereby claim:

  • I am rossjones on github.
  • I am rossjones (https://keybase.io/rossjones) on keybase.
  • I have a public key whose fingerprint is 8ADD 28B3 4EEB ACC7 1717 62FB 068D 0142 5095 D752

To claim this, I am signing this object:

"""
Converts the VOA Summary non-domestic rates data from a not-quite-csv to a huge Json blob.
python convert.py > summary.json < data/uk-englandwales-ndr-2017-summaryvaluations-proposed-epoch-0001-baseline-csv.csv
"""
import io
import json
import sys
<script type="application/ld+json">
{
"@context": "http://schema.org",
"@type": "Dataset",
"@id": "{% url_for 'dataset_read', id=c.pkg_dict.name, qualified=True %}",
"name": "{{ c.pkg_dict.title }}",
"description": "{{ c.pkg_dict.notes }}",
"catalog": {
"@type": "DataCatalog",
"@id": "{% url_for 'search',qualified=True %}"
'#name?',
'.expense description',
'0',
'1 department family',
'2 entity',
'3 date',
'4 expense type',
'5 expense area',
'6 supplier',
'7 transaction no',
@rossjones
rossjones / download_discovery_registers.rb
Created July 29, 2016 15:24 — forked from robmckinnon/download_discovery_registers.rb
Download discovery registers as TSV
`mkdir data`
%w[uk
street
school
school-type
school-tag
school-phase
school-gender
school-federation
school-admissions-policy
Interactive Elixir (1.2.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> spawn(fn-> :ets.new(:rooms, [:named_table, :set]) end)
#PID<0.59.0>
v(2): :ets.insert_new(:rooms, {"Test", []})
** (ArgumentError) argument error
(stdlib) :ets.insert_new(:rooms, {"Test", []})