Flyscanning data setup

README.md

Flyscanning data setup

Overview

1. Documents, including StreamResource and StreamDatum, are emitted by RunEngine.
2. New TiledWriter callback extracts information from documents and makes HTTP calls to Tiled, storing the locations and relative alignment of the files.
3. Export code uses Tiled as a "directory service" (borrowing Callum's term) to locate the files for a given scan. It packs up "NeXus file", in accordance with beamline requirements, by making h5py.ExternalLinks into the raw files.

At this time we are using Tiled only as a directory service, but of course it could also be used access sliced and transcoded data over HTTP or stream the whole raw files over HTTP.

Setup

Generate secret we can use:

openssl rand -hex 32

One-time setup:

mkdir /tmp/tiled_storage
tiled catalog init /tmp/tiled_storage/catalog.db  # create tables

With latest (v0.1.0a117) tiled:

tiled serve /tmp/tiled_storage/catalog.db -w /tmp/tiled_storage/data -r /path/to/where/iocs/write --api-key=<SECRET>

Then, in the Bluesky terminal, with the genematx:add-tiled-writer branch of bluesky:

export TILED_API_KEY=<SECRET>

In IPython or its startup scripts:

from tiled.client import from_uri
# needs Eugene's branch of bluesky:
from bluesky.callbacks.tiled_writer import TiledWriter

client = from_uri("http://localhost:8000")  # will use TILED_API_KEY from env
tw = TiledWriter(client)
RE.subscribe(tw)

Export

Simple sketch...

In [1]: from tiled.client import from_uri

In [2]: c = from_uri("http://localhost:8000", include_data_sources=True)  # uses TILED_API_KEY from env

In [3]: c['a1451ea2-55c5-4d45-a4c1-efc0872e4355']
Out[3]: <BlueskyRun {'panda_standard_det_stream', 'kinetix_standard_det_stream'} scan_id=184 uid='a1451ea2' 2024-03-28 20:11>

In [4]: c['a1451ea2-55c5-4d45-a4c1-efc0872e4355']['panda_standard_det_stream']
Out[4]: <Container {'external', 'internal'}>

In [5]: c['a1451ea2-55c5-4d45-a4c1-efc0872e4355']['panda_standard_det_stream']['external']
Out[5]: <Container {'hex-panda1-calc1-out-Value', ...} ~9 entries>

In [6]: c['a1451ea2-55c5-4d45-a4c1-efc0872e4355']['panda_standard_det_stream']['external']['hex-panda1-pcap-ts_trig-Value']
Out[6]: <ArrayClient shape=(1801,) chunks=((1024, 777),) dtype=float64>

Get the filepath(s) backing this data:

In [7]: from tiled.client.utils import get_asset_filepaths

In [8]: get_asset_filepaths(c['a1451ea2-55c5-4d45-a4c1-efc0872e4355']['panda_standard_det_stream']['external']['hex-panda1-pcap-ts_trig-Value'])
Out[8]: [PosixPath('/nsls2/data/hex/proposals/commissioning/pass-315051/tomography/bluesky_test/panda/748c016b-9b71-4ea4-9973-4847a5f877c3.h5')]

The above is convenience wrapper around this information-dump:

In [9]: c['a1451ea2-55c5-4d45-a4c1-efc0872e4355']['panda_standard_det_stream']['external']['hex-panda1-pcap-ts_trig-Value'].data_sources()
Out[9]: 
[{'id': 124,
  'structure_family': 'array',
  'structure': {'data_type': {'endianness': 'little',
    'kind': 'f',
    'itemsize': 8},
   'chunks': [[1024, 777]],
   'shape': [1801],
   'dims': None,
   'resizable': False},
  'mimetype': 'application/x-hdf5',
  'parameters': {'path': ['PCAP.TS_TRIG.Value']},
  'assets': [{'data_uri': 'file://localhost/nsls2/data/hex/proposals/commissioning/pass-315051/tomography/bluesky_test/panda/748c016b-9b71-4ea4-9973-4847a5f877c3.h5',
    'is_directory': False,
    'parameter': 'data_uri',
    'num': None,
    'id': 36}],
  'management': 'external'}]

Or read data directly via HTTP (probably not what you want, but just for the sake of a demo...)

In [10]: c['a1451ea2-55c5-4d45-a4c1-efc0872e4355']['panda_standard_det_stream']['external']['hex-panda1-pcap-ts_trig-Value'][:]
Out[10]: 
array([ 1.93255802,  1.94925298,  1.96595638, ..., 31.88089343,
       31.8975904 , 31.91406612])

Real working example, which uses get_asset_filepaths and makes h5py.ExternalLink: https://github.com/NSLS-II-HEX/workflows/pull/4/files