Skip to content

Instantly share code, notes, and snippets.

Avatar

Etienne Posthumus epoz

  • Amsterdam, Nederland
View GitHub Profile
@epoz
epoz / codesigning.md
Last active Feb 14, 2021
Code signing certificates prices/availability
View codesigning.md
@epoz
epoz / ndjson2nt.py
Created Jan 29, 2021
Scan a NDJSON file of JSON-LD data and convert it to Turtle
View ndjson2nt.py
# Make sure you have the rdflib and rdflib-jsonld libraries installed
# This gist is the Pythn equivalent of Mark's repo from here: https://github.com/mightymax/ndjson2ttl
# As referenced in this tweet: https://twitter.com/markuitheiloo/status/1355252255327449090
import sys
from rdflib import Graph
for line in open(sys.argv[1]):
g = Graph().parse(data=line, format="json-ld")
print(g.serialize(format="n3").decode("utf8"))
@epoz
epoz / scan_publishers.py
Created Apr 23, 2020
Scan Crossref data dump of 2020-04-14
View scan_publishers.py
import json
import gzip
import os
from progress.bar import Bar
# Scan the Crossref data dump as mentioned in : https://twitter.com/CrossrefOrg/status/1250146935861886976
# And parse out the publishers names, so you know where in the giant dump your own data can be found
# Note this script uses the progress library, so before running do a "pip install progress"
filenames = [filename for filename in os.listdir('.') if filename.endswith('.json.gz')]
@epoz
epoz / notes.md
Last active Jan 9, 2020
How to expose a port on a running Docker container
View notes.md

Getting to access a internal port inside a running Docker container

Now that we have moved to a mostly Docker-based infrastructure, one of the tricky things is to try and debug things when there is something pear-shaped. It used to be possible to just SSH into the machine with a local port-forward, and then for example access the Elasticsearch server via a handy browser extension to do debugging.

But what to do if your container is running in a Docker Swarm and has no ports forwarded by default? (which is the right thing to do, keep it simple and closed...) Thanks to stirling help from https://github.com/eelkevdbos here is the solution, and I am writing it up here so I can remember it in future, cause I sure am gonna forget the details...

First thing, create a new docker overlay network that you can use for getting to the container in question:

docker network create foobar

@epoz
epoz / dzi_to_iiif.py
Created Nov 8, 2019
Converts a directory of Deepzoom images to IIIF Level-0 compliant static store
View dzi_to_iiif.py
import os
import sys
if __name__ == '__main__':
dzifile = sys.argv[1]
for dirpath, dirnames, filenames in os.walk('.'):
for filename in filenames:
print(os.path.join(dirpath, filename))
@epoz
epoz / linkall.py
Last active Oct 17, 2019
HardLink all files into single directory
View linkall.py
#!/usr/bin/env python
u = """Link all files from a directory and its descendants into a specified destination directory.
This 'flattens' the source dir into the destination dir
Usage: %s source_dir destination_dir
"""
import os
import sys
from progress.bar import Bar
@epoz
epoz / thumbs.py
Created Apr 4, 2019
Read a directory of images and convert them into a single sqlite3 db of thumbs
View thumbs.py
# sqlite> .schema
# CREATE TABLE thumbs(filename, data);
from PIL import Image
import os
from io import BytesIO
import sys
import sqlite3
from progress.bar import Bar
View gist:e6f42fef31180e75f21ebed373a73fe2
### Keybase proof
I hereby claim:
* I am epoz on github.
* I am epoz (https://keybase.io/epoz) on keybase.
* I have a public key ASAX-cAXJOiDTnX1CA73U80bpWL-sbX1XNsQGvWVi3BLZAo
To claim this, I am signing this object:
@epoz
epoz / uniq_ic.py
Created Oct 24, 2018
Make Iconclass notations unique for a given textbase file
View uniq_ic.py
import iconclass
import textbase
import sys
from progress.bar import Bar
d = textbase.parse(sys.argv[1])
bar = Bar('Processing', max=len(d))
def is_in_there(notation, notations):
@epoz
epoz / dbtxt_to_csv.py
Created Oct 19, 2018
Dumps a collection of dbtxt files to a single CSV file, including text of IC field expanded
View dbtxt_to_csv.py
from __future__ import print_function
# Export details from a collection of dmp files found at a certain path to a csv file
import os
import iconclass
import sys
from progress.bar import Bar
import textbase
print('Reading files...')