Skip to content

Instantly share code, notes, and snippets.

View vsoch's full-sized avatar
💭
rawr

Vanessasaurus vsoch

💭
rawr
View GitHub Profile
@vsoch
vsoch / python-hashing.py
Created February 16, 2019 21:17
A quick way to run a single file through a bunch of hashing algorithms, via Python's hashlib
import hashlib
filename = '/home/vanessa/.singularity/shub/vanessa-marshmallow-latest@8dc77d81e1775986b89f042112c5ee81.sif'
for algorithm in hashlib.algorithms_guaranteed:
hasher = getattr(hashlib, algorithm)()
with open(filename, "rb") as f:
for chunk in iter(lambda: f.read(4096), b""):
hasher.update(chunk)
try:
print("%s: %s" %(algorithm, hasher.hexdigest()))
except:
@vsoch
vsoch / singularity_example_bash.sh
Last active February 4, 2019 16:49
An example script to run a Singularity container for a user on the list
#!/bin/bash
# Before you run this script, pull your container somewhere, and then provide
# the full path as an argument to this script
# If your $HOME is small, export to where you have more space like scratch
# export SINGULARITY_CACHEDIR=$SCRATCH/.singularity
# singularity pull --name antismash-standalone-4.2.0.sif docker://antismash/standalone:4.2.0
# Then provide the full container path as the last argument.
set -o errexit
@vsoch
vsoch / pull_static_registry.sh
Last active February 2, 2019 16:55
A quick example of how to pull a container from a static registry (https://singularityhub.github.io/sregistry-org)
# Define the registry base, the container URI, and the tag
REGISTRY=https://singularityhub.github.io/registry-org
CONTAINER=singularityhub/centos
TAG=latest
# Use curl and jq to parse the manifest and obtain the download link
STORAGE=$(curl "${REGISTRY}/${CONTAINER}/manifests/${TAG}/" | jq --raw-output '.layers [] .urls []')
DIGEST=$(curl "${REGISTRY}/${CONTAINER}/manifests/${TAG}/" | jq --raw-output '.layers [] .digest')
# Use Singularity to pull (here I'm using the Dockerized version, note it's pulling to $PWD)
@vsoch
vsoch / with-google-dns.sh
Created January 14, 2019 19:07
Singularity Hub Debugging
## The following shows some debugging output when Google nameservers are added to /etc/resolv.conf (8.8.8.8 and 8.8.4.4)
#################################################################
$ whois singularity-hub.org
Domain Name: SINGULARITY-HUB.ORG
Registry Domain ID: D402200000000429751-LROR
Registrar WHOIS Server: whois.google.com
Registrar URL: http://domains.google.com
Updated Date: 2019-01-14T16:27:45Z
@vsoch
vsoch / in_memory_tar_extract.py
Created December 12, 2018 23:52
A quick example of reading gzip archives into memory FROM a tar archive (mind blown!)
import sys
import tarfile
input_tar = sys.argv[1]
# If input tar is not found, do not proceed
if not os.path.exists(input_tar):
print('Cannot find %s!' % input_tar)
sys.exit(1)
@vsoch
vsoch / extract_tar.py
Created December 12, 2018 14:52
Neato! Extract a .tar.gz from a .tar, and read the file content, all in memory
import sys
import tarfile
input_tar = sys.argv[1]
tar = tarfile.open(input_tar, 'r')
for member in tar:
print("Extracting %s" % member.name)
subtar = tarfile.open(mode='r|gz', fileobj=tar.extractfile(member))
for submember in subtar:
@vsoch
vsoch / editTarfile.py
Last active November 14, 2018 01:43
example of opening and editing a .tar.gz (change permissions in this case)
import tarfile
import tempfile
import stat
import os
tar_file = "input.tar.gz"
tar = tarfile.open(tar_file, "r:gz")
members = tar.getmembers()
file_permission = stat.S_IRUSR | stat.S_IWUSR
@vsoch
vsoch / opensource-cs.md
Last active August 12, 2018 15:18
Why Open Source + Computer Science Education should be a thing

The Open Source Lab

It's hard to not notice that training in computer science has been streamlined to primarily fill industry-level engineering jobs. In this respect, a degree from a top university in computer science is a lucrative choice for a young person to make. The path, and resulting enterprise tools that result from this pipeline are impressive - advances in machine learning, GPUs, and mobile technologies that are changing the world. However, in this overwhelming positivity, it's easy to look over the empty space, or specifically, to ask where those same graduates aren't going.

As a research software engineer, I came out of traditional academia, and first hand

@vsoch
vsoch / transfer_notebook.md
Last active September 23, 2019 09:48
a quick tutorial to use the forward tool with sherlock/py3-jupyter to transfer a notebook first

The user issue was asking how to work with notebooks that are on the host. This is the response, associated with this release


Let's just make sure we are working from the same thing. Make sure your forward repository is up to date with the latest on Github, and that you have run setup.sh so that there is a CONTAINERSHARE and RESOURCE variable in your params.sh

USERNAME="vsochat"
PORT="43453"
PARTITION="russpold"