Skip to content

Instantly share code, notes, and snippets.

View jakirkham's full-sized avatar

jakirkham

247 followers · 55 following
View GitHub Profile
@jakirkham
jakirkham / AOLLSM_conv_Zarr.ipynb
Last active May 24, 2019 16:54
Brief experiment loading the AOLLSM dataset
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@jakirkham
jakirkham / gtree.py
Created May 16, 2019 15:09
Google Drive utility functions
from itertools import chain, tee
def walk(drive, root_dir, maxResults=10):
dirs = iter([(root_dir["title"], root_dir)])
while True:
try:
cur_path, cur_dir = next(dirs)
except StopIteration:
break
@jakirkham
jakirkham / dict_code_mvpy.py
Last active August 29, 2018 17:36
Helper script for generating dict/code movies
import dask.array as da
import zarr
fn = "<some filename>"
fh = zarr.open_group(fn + ".zarr")
dict_ = da.from_array(fh["dict"], chunks=fh["dict"].chunks)
dict_min = dict_.min()
@jakirkham
jakirkham / numpy_pickle_protocol_5.py
Created August 1, 2018 04:48 — forked from ogrisel/numpy_pickle_protocol_5.py
Draft use of pickle protocol 5 (PEP 574) for zero-copy numpy array pickling
from pickle5 import Pickler, load, PickleBuffer
import copyreg
import os
import numpy as np
import time
import gc
import io
from multiprocessing import get_context
@jakirkham
jakirkham / isfinite.pyx
Last active July 25, 2018 21:19
Cython implementation of isfinite as a reduction over NumPy arrays
cimport libc
cimport libc.math
from libc.math cimport isfinite as _isfinite, isinf as _isinf
cimport cython
cimport numpy
import numpy
numpy.import_array()
@jakirkham
jakirkham / shm.py
Last active January 20, 2024 15:27
Some Python ctypes-based POSIX shared memory functions
import ctypes
import mmap
import os
import stat
import sys
try:
unicode
except NameError:
@jakirkham
jakirkham / shmemctypes.py
Created June 23, 2018 15:00 — forked from shawnchin/shmemctypes.py
shm_open() version of multiprocessing.sharedctypes.RawArray
#
# Based on multiprocessing.sharedctypes.RawArray
#
# Uses posix_ipc (http://semanchuk.com/philip/posix_ipc/) to allow shared ctypes arrays
# among unrelated processors
#
# Usage Notes:
# * The first two args (typecode_or_type and size_or_initializer) should work the same as with RawArray.
# * The shared array is accessible by any process, as long as tag matches.
# * The shared memory segment is unlinked when the origin array (that returned
@jakirkham
jakirkham / dask_example_copy_access.py
Last active June 1, 2018 17:17 — forked from hanslovsky/dask_example.py
Show dask array in BigDataViewer
import ctypes
import dask.array
import dask.array.image
import dask.array as da
import numpy as np
import os
import pathlib
import skimage.io
import subprocess
import sys
@jakirkham
jakirkham / distributed_shutdown_challenges.ipynb
Last active May 7, 2018 02:37
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@jakirkham
jakirkham / lsf_janelia_drmaa.sh
Last active March 30, 2018 20:31
# Export LSF variables, if available.
# May not be available when using Linux locally or Windows with Git Bash.
if [[ -f /misc/lsf/conf/profile.lsf ]]; then
# Needed to activate LSF and specify DRMAA's location.
source /misc/lsf/conf/profile.lsf
export LSF_DRMAA_LIBRARY_PATH=/misc/sc/lsf-glibc2.3/lib/libdrmaa.so.0.1.1
export DRMAA_LIBRARY_PATH=$LSF_DRMAA_LIBRARY_PATH
# Other nice options with LSF, but not required.
export LSB_STDOUT_DIRECT='Y'
export LSB_JOB_REPORT_MAIL='N'