2.md

Controlling parallel code through functional programming

In the previous article we looked at sequential asynchronous code and how to tame it using functional programming.

We also introduced how to use asynchronous function composition using composite and partial function application using ap to reduce complexity. We will be re-using those techniques so I recommend you read the previous article and familiarize yourself with them

Parallel code

Today we will look at parallel code and how we can use asynchronous iterators to manage their complexity

An example of parallel code would be

fs.readdir("/tmp", function (err, fileNames) {
    var count = files.length
        , files = []

    for (var i = 0; i < files.length; i++) {
        fs.readFile(files[i], function (err, file) {
            if (err) {
                return next(err)
            }

            files.push(file)
            count--

            if (count === 0) {
                next(null, files)
            }
        })
    }

    function next(err, results) {
        // results is an array of files
        // do something
    }
})

The above code reads all the files in the /tmp folder and then calls next with an array of the file content.

What it's actually doing is two actions, one is reading the directory and the second is mapping all the file names to the actual file content.

Although it's not technically mapping the array because fs.readFile is asynchronous and returns in arbitrary order. We are using files.push(file) which will just turn files into an array of the files in random order. This is an easy mistake to make.

Parallel map

This means we can represent this code as a composition of reading the directory and mapping the files to their content.

var action = composeAsync(
    mapToFiles
    , fs.readdir
)

action("/tmp", doSomething)

function doSomething(err, results) {
    // do something
}

function mapToFiles(err, fileNames, callback) {
    iterators.map(fileNames, fs.readFile, callback)
}

In this case iterators.map takes a list (either an object or array) and calls an iterator function (in this case fs.readFile) for each value in the list (each fileName in the array).

iterators assumes your iterator function applies to the standard asynchronous syntax of callback last and that the callback is called with (err, data). When we write code that complies with this standard node pattern it simplifies a lot of the complexity.

iterators.map itself will take the result value from the callback and replace the initial value in the list with it. iterators.map also keeps a running count of whether all the asynchronous functions are finished; when they are iterators.map will call the third parameter - your done callback -with the mapped list.

For more details read the implementation of iterators.map

iterators.map itself will take the result value from the callback and replace the initial value in the list with it. iterators.map also keeps a running count of whether all the asynchronous functions are finished and when they are it will call the third parameter which is your done callback with the mapped list.

iterators.map itself will take the result value from the callback and replace the initial value in the list with it. iterators.map also keeps a running count of whether all the asynchronous functions are finished; when they are iterators.map will call the third parameter - your done callback - with the mapped list.

I understand the reasoning behind this approach is to avoid a control flow lib, however what makes using these libraries not a control flow lib "suite" when used all together in order to tackle the control flow problem? I suppose the argument is that using these libs you have more granular functionality and could use them for other purposes beside control flow, but is this the only argument you are making for using these libs, or even one at all?

@samholmes

It's not about flow control.

It's about realizing what you really want to do is asynchronously map one list to another. Which is a simple concept.

flow control is a vague and hand wavey concept with weird constructs in them.

Functional programming is simple and low in complexity in comparison.