kharandziuk

This article shows how to apply Node.js Stream and a bit of Reactive programming to a real(tm) problem. The article is intended to be highly practical and oriented for an intermediate reader. I intentionally omit some basic explanations. If you miss something try to check the API documentation or its retelling(e.g.: this one)

So, lets start from the problem description. We need to implement a simple web scraper which grabs all the data from some REST API, process the data somehow and inserts into our Database. For simplicity, I omit the details about the actual database and REST API(in real life it was the API of some travel fare aggregator website and a Pg database)

Consider we have two functions(code of the IO simulator functions and the other article code is here):

getAPI(n, count) // pseudo API ca

darsain

Better streaming interface for modern JavaScript

Node streams have a lot of issues. Awkward to control backpressure, no error propagation, overcomplicated implementation impenetrable to any view source attempts, etc...

To solve this, here is an implementation of pull-streams in modern JS, using promises, async iterators, and for..await loops.

Features:

• Built in backpressure.
• Build in error propagation.