Skip to content

Instantly share code, notes, and snippets.



Created Mar 23, 2017
What would you like to do?
Real-time Insights powered by Reactive Programming

Real-time Insights powered by Reactive Programming

Presenter: Jay Phelps (@_jayphelps)

  • Jay Phelps
    • Senior Software Engineer
  • InfoSec
    • Preventing unauthorized access
    • Stopping hackers
    • Can block exploits using their gateway proxy
      • They need to know if it's working
      • They want to watch attackers try
        • to learn/refine their defenses
        • they want to see it in real-time
  • Real-time
    • for debugging, infosec, logging
    • thousands of servers, millions of devices, all logging
    • how do they handle massive amounts of streaming logs?
  • RX
    • best ideas from observer pattern, iterator pattern, and functional programming
    • "lodash for events"
    • Available for many languages
    • FB, Slack, Microsoft, Netflix, Google, GitHub, Airbnb
    • High level intro
      • Observable = basic primitive
      • Array = collections of item
      • Observable = collections of items over time
      • Represents a stream
      • Marble diagrams are useful for visualizing how thing work with streams (things over time)
      • Stream of logs
    • Angular 2 uses RxJS
  • Logging
    • Logging JSON
    • Created a SQL like query language to create RxJava
    • 8 million messages per second, at peak: how do we scale this?
      • Load balancing jobs (autoscaling)
      • Need to chain jobs together
      • Segment traffic, then query
      • High-volume distributed systems have a problem: backpressure
        • pressure opposed to the desired flow
        • deficit of ability to calculate, causing set of things unable to be handled, growing over time
        • Can Buffer (hold and wait) or Drop (discard)
      • Job authors get to choose whether to drop or buffer
        • they buffer, then autoscale, then drop
    • Netflix's lib for this is Mantis
    • Where do they actually query this?
      • Query builder UI
        • Source, Fields, Conditions, and Query Preview
      • Problem: can be really high volume (100k+ rps)
        • Performance solutions are often driven by UX
        • partial Solution: UI virtualization
          • still can't update the virutal table 100k/s
          • this inability to update is also backpressure
          • need to buffer or drop
          • UX problem still not resolved; buffer size is unbounded
      • Users just want a sample
        • See a little bit, in real time, then refine the query
        • solution: batch sampling
          • Buffer for a bit, then drop after reaching a certain threshold
          • 50 was a great number
          • How this is implemented:
          • Rx enabled making this really easy (and performantly)
          • Works for low-volume queries too
            • If im a tester, on a device, I just want to see my requests
            • I can turn on the firehose if I want
        • This tool is called Raven at Netflix
          • (It's beautiful) - Ryan
          • Can write Javascripts
          • Can push transformations to the servers
          • Has alerting!
          • Has improved debugging, testing, and InfoSec
    • Netflix loves Rx
      • It's powerful, and cross-platform
      • Can follow you to basically any language
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.