dylanmcreynolds

# Source:
# https://www.cloudflare.com/ips
# https://support.cloudflare.com/hc/en-us/articles/200169166-How-do-I-whitelist-CloudFlare-s-IP-addresses-in-iptables-

for i in `curl https://www.cloudflare.com/ips-v4`; do iptables -I INPUT -p tcp -m multiport --dports http,https -s $i -j ACCEPT; done
for i in `curl https://www.cloudflare.com/ips-v6`; do ip6tables -I INPUT -p tcp -m multiport --dports http,https -s $i -j ACCEPT; done

# Avoid racking up billing/attacks
# WARNING: If you get attacked and CloudFlare drops you, your site(s) will be unreachable. 
iptables -A INPUT -p tcp -m multiport --dports http,https -j DROP

dylanmcreynolds

This document is very much a work in progress.

Overview

Databroker projections provide a method to map from beamline-specific Databroker structures and names to common datastructures and names. They do this through a mapping document (a (projection)[https://github.com/bluesky/event-model/blob/8be9bd49b0ce76b64a9a111c1091364beedb1671/event_model/schemas/run_start.json#L8]) and python code that takes a BlueskyRun and a Projection as inputs, and returns a an xarray.Dataset that maps fields and values from the BlueskyRun.

Projections inherintly perform two separate forms of mapping: structural and semantic.

Projections and Semantic Mapping

Projections map field names from one ontology to the other. If RunA were to define an array of "whites" and another were to define an arrary of "brights", the each run could have a separate Projection that maps to those fields to a projected field called "flats".

dylanmcreynolds

Authorization Thoughts

Data acquired at BES light sources is being stored in a variety of locations with a variety of policies touching on retention, storage, access and requirements to make data public.

More facilities are implementing instances of Bluesky Data Broker, which serves as a data access framework for raw beamline data. Based on up-coming work, analyzed data will also be stored in Data Broker, with an association stored that links analyzed data sets with their raw collected sets. The data is stored in several formats, from mongodb to data files outside of mongodb.

Data Broker provides a framework that eases the access of data and metadata for beamline experiments. Bluesky can store beamline data directly in mongodb, and can reference external files collected by detectors, providing users with a unified view of their data with an API that can support multiple analysis tools. It does not, however, provide any built-in enforcement of access controls. One could attempt to control access directly t

dylanmcreynolds

The current Bluesky Databroker browser implementation in Xi-Cam works roughly like this:

1. On initialization, a query is made to databroker to get all BlueskyRuns, with a max return value set in the constant MAX_SEARCH_RESULTS (currently 100).
2. A cache is kept of all UIDs that have been loaded. When a BlueskyRun that has not been displayed is detected, it's contents are extracted from the BlueskyRun object and put into a variable (search_state._new_entries). show_results() is then triggered and runs on the MainThread. It extracts data from _new_entries and adds them to the table's Model class.
3. Every 10 seconds, a thread in the background re-runs the query. New items are added to search_state._new_entries and are then processed and added to the model.
4. When a databroker query finds more than 100 runs, currently there is no way to see beyond the initial 100 without applying a more advanced filter in the UI. Scrolling to the bottom does not re-issue a request.

The initial load of the catalog has been o