How to combine the Flux pattern with large hierarchical data

problems with Flux.md

How to combine the Flux pattern with large hierarchical data

I've been converting the web client part of an application I'm developing for a client from plain old JavaScript to React and Flux. (Because this is closed-source client work, I will be a bit vague about what the application is and does.)

Something about my data, my use case, and how I've implemented things, or some combination of those three, doesn't fit the Flux pattern well. So I am writing this to explain my problem and hopefully get some interesting feedback on it. (If you're reading this: thanks.)

The application

I have a Python server which talks to a PostgreSQL database and provides data in JSON format over a REST-ish API (not pure REST, more RPC-style).

I have a web application, written in JavaScript, which allows the user to browse and visualize that data.

The data

The data consists of data recordings - things that happened inside a game. It is structured as follows:

• There are multiple recordings.
• Each recording consists of multiple frames.
• Each frame contains multiple 'bbnodes'. A large amount of them, in fact.
• Each frame contains multiple 'entities'.
• Each frame contains multiple 'btnodes' per entity.

(I know 'bbnode' etc. is vague, but I believe the details don't matter.)

As a diagram:

    recordings
        -> frames
            -> bbnodes
            -> entities
                    -> btnodes

Size matters

One very important thing about the data: there is a lot of it, since it's coming from a 30 FPS game. So much we don't want to load it all into memory at once. This leads to a lot of decisions regarding the server API, the client architecture, etc. Right now maybe the data might fit most of the time, but we know the size of the data is only going to grow, so this is an assumption I don't want to reject.

The interaction flow

• You select a recording.
• You then select a frame within that recording.
• You can then browse the bbnodes within that frame.
• You can also select an entity within that frame, after which you can browse the btnodes of that entity (of that frame of that recording).

When you change the frame, all selections try to maintain themselves. So if you selected entity 'Jim' in frame 23, and you go to frame 790, the client will try to re-select 'Jim' if it can. This is important for a smooth user experience.

Summary

• I have a LOT of data.
• It rarely changes.
• When it does, it’s the server that changes it (through websockets).

Questions and problems

Where do I store the selection?

For some reason I have never seen examples or code that handle the kind selection the way I do it, and I'm not sure why. Either I am doing it wrong, or I haven't found the examples, or I have a weird use case.

In any case, I store selection, as in the data that indicates what is selected, in Flux Stores. This is because a ton of my UI depends on which recording/frame/entity is selected, if any. ('Nothing' is a very valid selection in every case.)

Maybe you select a frame, and that frame has no entities. Then I want to see 'There are no entities' in this frame on screen. Or, if you had an entity selected, I want that same entity's btnode UI to be updated.

I not only want UI to be re-rendered when the selection changes, I need to load additional data as well. I don't really see a way around this - you select frame 37, I need to show you the data for frame 37, that data is not in memory, so I need to load it.

Because various different parts of the UI need to update and I need to load data when the selection changes, I store that selection in the corresponding store. RecordingStore contains selectedRecording, RecordingFrameStore contains currentFrameNr, etc.

I have never seen anyone else do this, I don't quite see how else to do it, but it has a ton of implications.

When do I load data?

I have discussed this in some detail here. Right now, I load data this way:

• The user interacts with a component, e.g. a drop-down list.
• The component's event handler calls an action creator, which dispatches (for example) a SELECT_RECORDING action.
• The RecordingFrameStore (!) reacts to this action and makes an AJAX call (through an API module).
• When the AJAX promise resolves, the RecordingFrameStore calls another action creator, which dispatches a RECEIVE_FRAME_DATA action.
• The RecordingFrameStore reacts to this action, updates its internal state, and emits a change event.
• Components (container components) react to this event, re-get data from stores, and call setState.
• React re-renders these components (or not).

Pros and cons of that in the article I linked to.

Cascading store updates and highly interdependent store code

The biggest problem I have right now is that cascading store updates (verboten in Facebook's implementation - it won't let you dispatch actions from actions) are inevitable, like so:

• The user selects a recording.
• The selector component dispatches a SELECT_RECORDING action.
• The RecordingStore's dispatch callback changes its selection and emits a change.
• The RecordingFrameStore's dispatch callback waits for the RecordingStore, then makes an AJAX call.
• When the AJAX call returns, the RecordingFrameStore dispatches RECEIVE_FRAME_DATA with the AJAX response.
• The RecordingFrameStore's dispatch callback takes the AJAX response and stores it.

And this, multiplied by five or so, because of all the stores that depend on each other. Every Store reacts to the actions of its parent Stores, and has to do things based on what it knows about what a parent Store will do.

All because I cannot have one store update itself based on a change in another store. At least, not with Facebook's Flux implementation - I believe there are other implementations that allow store dependencies.

I don't know how to solve this, but here are some possibilities:

• All the selections go into one SelectionStore.
• All the data goes into one big Store.
• Flux is not the right pattern here.
• Facebook's Flux implementation is not the right implementation.
• I'm doing it wrong, or looking at the problem wrong.

Every entity in your app needs to have a global ID.
You don't need to refill the stores when selection changes if this is the case.

You'll just fetch the missing entities so stores consume them (and update ID -> entity maps). "Parent" stores will just update ID arrays corresponding to parent entity.

If selected author changes, fetch the new author and the new books for it. Have author entity in AuthorStore contain an array of book IDs, and have new books in BookStore stores by their IDs. This prevents the need for any nested updates: entires are added to "bags" instead of replaced. Think database tables.

I describe this approach in greater detail here: https://github.com/gaearon/flux-react-router-example

Here is the library I wrote to normalize nested API responses for easier consumption by independent stores: https://github.com/gaearon/normalizr

Flux is only hard if your data isn't normalized. Treat your Stores as database tables, use global unique IDs to reference related entities, don't clear data on UI change. Then Flux will be easy to work with.

Here's a good rule of thumb: you should never need to delete entities from the store. When selection changes, only the selected IDs change. They may even be kept in a separate SelectionStore.

This also lets you conveniently show the old data before the new data has arrived by keeping previousSelectedAuthorId and selectedAuthorId and show the one that is loaded.

The data never changes, unless a new recording appears. It's only the selection that changes.

I can conceptualize my data as an array of arrays, holding tree structures to big too fit into memory, loaded opportunistically.