Bluesky Xi-Cam Browser List Enhancements

gistfile1.md

The current Bluesky Databroker browser implementation in Xi-Cam works roughly like this:

1. On initialization, a query is made to databroker to get all BlueskyRuns, with a max return value set in the constant MAX_SEARCH_RESULTS (currently 100).
2. A cache is kept of all UIDs that have been loaded. When a BlueskyRun that has not been displayed is detected, it's contents are extracted from the BlueskyRun object and put into a variable (search_state._new_entries). show_results() is then triggered and runs on the MainThread. It extracts data from _new_entries and adds them to the table's Model class.
3. Every 10 seconds, a thread in the background re-runs the query. New items are added to search_state._new_entries and are then processed and added to the model.
4. When a databroker query finds more than 100 runs, currently there is no way to see beyond the initial 100 without applying a more advanced filter in the UI. Scrolling to the bottom does not re-issue a request.

The initial load of the catalog has been observed to take an unacceptable amount of time (5-20 seconds at real beamlines). This is especially annoying to users as Xi-Cam is not giving the user a visual indication that processing is happening. Analysis with brute-force logging has failed to show a single culprit. The initial load of 100 runs simply takes a long time with all of the processing. Here, we discuss ways to address this.

Cheap fix

Reduce MAX_SEARCH_RESULTS to, say 20. This will decrease the initial time-to-display of runs. There are better fixes, but this might be good enough for the short term.

Advanced Fix

In addition to reducing the initial display size, we can implement lazy loading. We could use QAbstractItemModel instead of QStandardItemModel for the SearchResultsModel. This allows for lazy loading by implementing canFetchMore() and fetchMore() so that when a user scrolls to or near the bottom of the list, we can issue small queries to populate another 10 (or so) runs. This would provide a faster initialization time, saving processing for when the user indicates that they need more runs.