Skip to content

Instantly share code, notes, and snippets.

@ssomnath
ssomnath / NSID.rst
Last active May 19, 2020
Data format for storing N-dimensional data in HDF5
View NSID.rst

N-Dimensional Spectroscopy and Imaging Data (NSID) Format

Preface

Several years ago, when I was a postdoctoral researcher at CNMS, a few fellow researchers and I explored multiple methods for representing measurement data. Long-story short, we decided to go ahead with what is now the Universal Spectroscopy and Imaging Data (USID) model where the data would be written into Heirarchical Data Format (HDF5) files. USID's ability to express all kinds of data, especially niche cases like compressed sensing, spiral scans (neither polar nor cartesian grid), etc., renders itself perhaps unnecessarily complicated for data which do have an N-dimensional form. When data does have a clear N-dimensional form, one could leverage a lot of HDF5's inherent capabilities to easily represent and use N-dimensional datasets. For this reason, I had explored the possibility of using USID only when an N-dimensional fo

@ssomnath
ssomnath / usid_hs.py
Created Apr 29, 2019
Utilities to read and write to USID HDF5 files from HyperSpy Signal objects
View usid_hs.py
import os
from warnings import warn
from functools import partial
import collections
import h5py
import numpy as np
import dask.array as da
from hyperspy.signals import BaseSignal, ComplexSignal
import pyUSID as usid
@ssomnath
ssomnath / find_string_in_directory.py
Last active Mar 29, 2020
Finds locations of all occurences of a string within a directory or code project
View find_string_in_directory.py
import os
def search_in_file(file_path, target_string, match_case=False, abs_paths=False):
if not match_case:
target_string = target_string.lower()
_, file_name = os.path.split(file_path)
with open(file_path) as myFile:
try:
for num, line in enumerate(myFile.readlines()):
if not match_case:
@ssomnath
ssomnath / json_from_h5_groups.py
Created Aug 30, 2018
Extract JSON metadata from all HDF5 groups
View json_from_h5_groups.py
from __future__ import print_function, division, unicode_literals
import os
import numpy as np
import h5py
import json
import sys
if sys.version_info.major == 3:
unicode = str
@ssomnath
ssomnath / lines_in_project.py
Last active Jan 23, 2020
Count lines in python project
View lines_in_project.py
import os
from collections import OrderedDict
def count_doc_lines(file_path, chars_per_line=120):
non_blank_count = 0
total_lines = 0
with open(file_path) as file_handle:
for line in file_handle:
curr = 1
if line.strip():