danielballan

Flyscanning data setup

Overview

1. Documents, including StreamResource and StreamDatum, are emitted by RunEngine.
2. New TiledWriter callback extracts information from documents and makes HTTP calls to Tiled, storing the locations and relative alignment of the files.
3. Export code uses Tiled as a "directory service" (borrowing Callum's term) to locate the files for a given scan. It packs up "NeXus file", in accordance with beamline requirements, by making h5py.ExternalLinks into the raw files.

At this time we are using Tiled only as a directory service, but of course it could also be used access sliced and transcoded data over HTTP or stream the whole raw files over HTTP.

danielballan

import json
from pathlib import Path

from bluesky.callbacks.tiled_writer import TiledWriter
from tiled.client import from_uri

documents = json.loads(Path("documents.json").read_text())
client = from_uri("http://localhost:8000", api_key="secret")
tw = TiledWriter(client)

danielballan

import dask
import dask.array as da
import distributed
from distributed.protocol.serialize import dask_deserialize, dask_serialize
from dask.distributed import Client
import h5py
import numpy as np

import logging
logger = logging.Logger(__name__)

danielballan

Example of how to extend Tiled with a custom route

1. Clone this gist and enter the directory.

2. Install Tiled and and, optionally, a HTTP client to test with.

   pip install tiled[all] httpie
   

danielballan

from ophyd.sim import Signal
from ophyd import Component, Device


class Filters(Device):
    a = Component(Signal, value=0)
    b = Component(Signal, value=1)

    def set(self, a, b):
        return (

danielballan

import metadatastore.conf
from collections import deque
from tqdm import tqdm
import ipyparallel as ipp


def main():
    old_config = dict(metadatastore.conf.connection_config)
    new_config = old_config.copy()
    new_config['database'] = 'metadatastore_production_v1'

danielballan

Tasks

1. Run python middlware.py.
2. Using HTTPie, request data: http :8000/data. This should return the list of numbers from the server.
3. Add a simple middleware that just uses print to print a message in the server logs before and after a request is seen.
4. Add a second middleware that also prints.

Question: Do the middlewares get run in the order that they are added in middlware.py or in the reverse order?

danielballan

Experimental Databroker Sandbox

1. Install podman. https://podman.io/getting-started/installation

2. Install podman-compose:

   pip install podman-compose

3. Clone or copy docker-compose.yml and config.yml from this gist into a directory.

danielballan

import argparse
import ftplib
import os
import tqdm
import time

"""
Example usage as a CLI:

```

danielballan

conda activate sandbox
pip install --pre tiled[all] databroker[all] ipython

Note for Mac: Mac uses zsh by default and it requires single quotes like 'databroker[all]'. On the other hand, Windows requires that you not have quotes.

mkdir sandbox
cd sandbox/