Skip to content

Instantly share code, notes, and snippets.

What would you like to do?
Discussion in regarding the scalability of Lightning in a production environment
deardooley 00:13
Hi guys, does anyone have experience running this in production use cases? What kind of performance considerations should be thought through prior to loading the service up? What are some practical constraints on the datasets the server can process and store?
mathisonian 00:42
hey @deardooley, i think it really depends on what scale we are talking about. in general setting ```NODE_ENV=production``` and configuring it to use a postgres database will do some performance optimizations for production use
at this stage the basic constraints would be data ingesting / and storage, again depending on your use case
it may be enough that a beefy postgres instance is fine with what you need to throw at it, or other backends could be explored
there are a few ways to scale out the server itself to if the amount of requests going in/out becomes the actual issue
im happy to discuss anything in more detail, lmk
deardooley 00:54
I’m wondering about the dataset size and how/what persistence looks like on the backend. If I have a dataset of 10-20M points, is that going to choke things? Is Pg the bottleneck? Could I run the API separately and just scale it out horizontally assuming we had elastic backend performance?
mathisonian 00:57
yeah, that should work. the basic assumption at this point is that you would never be trying to visualize a crazy number of points in a single graph, but you may have a large dataset across many visualizations
and then yes you could just scale horizontally in that case
deardooley 00:59
For context, I’m looking at lightning as a potential technology to add to the Agave Platform. Scientists love them some viz, but it’s also important that we find helpful ways to visualize WHAT they are doing. For example, dynamically generating charts of their application, network, lab performance from any event. 10’s of millions of events, but probably just 10’s of thousands of charts daily.
Are there any recent load numbers you’ve run as part of your release testing?
mathisonian 01:00
ah cool i dont see any hangups beyond basic horizontal scaling there
deardooley 01:01
and scaling is stateless, correct? I can just fire up another container, add it to the load balancer, and it’s good to go?
mathisonian 01:01
the load testing we've been doing has been performance of large sets into individual visualizations in the browser so i dont think they correspond exactly
yeah, for your case its should be like standard scaling of any nodejs/express/postgres app
deardooley 01:02
that’s helpful too. the geneticists and neuro guys will choke a memory card or two.
mathisonian 01:03
they are trying
deardooley 01:03
Pg because you’re familiar with it, or because you wanted triggers, views, etc? Is there a hangup backing it by Mongo or Redis?
mathisonian 01:03
we rely on some of its json features
but there are wrappers included if you want to use a different DB
we use sqlite for simple stuff
its using the sqlize orm + a few things to make it more performant with postgres
deardooley 01:08
kk. will try to take a look at the compose build and get back to you on experiences prototyping it for some friendly users.
mathisonian 01:09
awesome! let me know if you run into any issues
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.