Skip to content

Instantly share code, notes, and snippets.

@tabatkins
Last active Mar 5, 2022
Embed
What would you like to do?
Comparing the three pipeline proposals

We've been deadlocked for a while on the pipeline operator proposal, for a few reasons. Partially it's just low implementor interest, as this is fundamentally just syntactic sugar, but also because there are three competing proposals and the proponents of each haven't been convinced by the others yet.

In this essay I hope to briefly outline the problem space, summarize the three proposals, and talk about what's gained/lost by each of them. (Spoiler: they're all nearly identical; we're arguing over very small potatoes.)

What is the Pipeline Operator?

When you call a JS function on a value, there are currently two fundamental ways to do so: passing the value as an argument (nesting the functions if there are multiple calls), or calling the function as a method on the value (chaining more method calls if there are multiple).

That is, three(two(one(val))) vs val.one().two().three().

The first style, nesting, is generally applicable - it works for any function and any value. However, it's difficult to read as the nesting increases: the flow of execution moves right-to-left, rather than the left-to-right reading of normal code execution; if there are multiple arguments at some levels it even bounces back and forth, as your eyes jump right to find a function name and left to find the additional arguments; and editting the code afterwards can be fraught as you have to find the correct place to insert new arguments among many difficult-to-distinguish parens.

The second style, chaining, is only usable if the value has the functions designated as methods for its class. This limits its applicability, but when it applies, it's generally more usable and easier to read and write: execution flows left to right; all the arguments for a given function are grouped with the function name; and editting the code later to insert or delete more function calls is trivial, since you just have to put your cursor in one spot and start typing or delete one contiguous run of characters with a clear separator.

The benefits of method chaining are so attractive that some very popular libraries contort their code structure specifically to allow more method chaining. (jQuery being the elephant in the room, as it's still the most popular JS library in the world, and its core design is a single über-object with a jillion methods on it, all of which return the same object type so you can continue chaining.)

The Pipeline operator attempts to marry the convenience and ease of method chaining with the wide applicability of function nesting. The general structure of all the pipeline operators is val |> one() |> two() |> three(), where one, two, and three are all ordinary functions that take the value as an argument; the |> glyph then does some degree of magic to "pipe" the value from the LHS into the function.

The three pipeline proposals just differ slightly on what the "magic" is, and thus on precisely how you spell your code when using |>.

Proposal One: F#-style

In this proposal, matching the F# language's pipeline syntax, the RHS of the pipeline is an expression that must resolve to a function, which is then called with the LHS as its sole argument. That is, you write val |> one |> two |> three to pipe val thru the three functions; the general syntax transform is left |> right => right(left).

Pro: The restriction that the RHS must resolve to a function lets you write very terse pipelines when the operation you want to perform is already a named function.

Pro: Only one new bit of syntax needs to be minted, the |> itself. (The others require a placeholder syntax as well.)

Con: The restriction means that any operations that are performed by other syntax must be done by wrapping the operation in an arrow func: val |> x=>x[0], val |> x=>x.foo(), val |> x=>x+1, val |> x=>new Foo(x) etc. Even calling a named function requires wrapping, if you need to pass more than one argument: val |> x=>one(1, x).

Con: The yield and await keywords are scoped to their containing function, and thus can't be handled by the arrow-func workaround from the previous paragraph. If you want to integrate them into pipeline at all (rather than requiring the pipeline to be paren-wrapped and prefixed with await), they need to be handled as a special syntax case: val |> await |> one to simulate one(await val), etc.

Proposal Two: Hack-style

In this proposal, matching the Hack language's pipeline syntax, the RHS of the pipeline is an expression containing a special placeholder variable, which is evaluated with the placeholder bound to the LHS value. That is, you write val |> one(#) |> two(#) |> three(#) to pipe val thru the three functions; the general syntax transform is left |> right(#) => right(left).

Pro: The RHS can be any expression, and the placeholder can go anywhere any normal variable identifier could go, so you can pipe to any code you want without any special rules at all: val |> one(#) for functions, val |> one(1, #) for multi-arg functions, val |> #.foo() for method calls (or val |> obj.foo(#), for the other side), val |> # + 1 for math, val |> new Foo(#) for constructing, val |> await # for awaiting promises, etc.

Con: If all you're doing is piping thru already-defined unary functions, it's slightly more verbose than F#-style since you need to actually write the function call syntax, adding a (#) to it.

Proposal 3: "Smart Mix"

In this proposal, mixing the previous two proposals, the RHS of the pipeline is either an expression containing a special placeholder variable, which is evaluated with the placeholder bound to the LHS value, or a bare (dotted-)ident which must resolve to a function, which is called with the LHS as its sole argument. That is, you write val |> one |> # + 2 |> three.foo to pipe val thru the expressions; the general syntax transform is either left |> right(#) => right(left) or left |> right => right(left).

Pros: It can handle all Hack-style cases with identical syntax, and additionally can handle most F#-style cases with identical syntax.

Con: If you're doing any higher-level programming to produce a function which'll eventually get called, you do need to use the placeholder; you can't rely on F#-style evaluation. That is, if you have a logger(fn) function that modifies a function to log its args and return value, you must pipe it like val |> logger(one)(#); writing val |> logger(one) is a syntax error. (This avoids garden-path issues, where it's not clear which syntax you're dealing with until you either spot the # or establish that there isn't one. Instead, the F#-style cases are trivial to spot because they're so restricted in syntax, and all the rest are Hack-style.)

Con: As a result of trying to split the difference between the two syntaxes, it ends up with a more complex mental model. If a pipeline RHS is currently written in F#-style, and you realize you want to do something else to it, you have to recognize that it must be shifted to Hack-style syntax. (Luckily it's a syntax error if you forget, rather than just evaluating to the wrong thing, so at least it's easy to recognize after-the-fact that this is needed.)

What's The Difference?

Ultimately... very little.

As long as F#-style has the special syntax forms for await and yield, then all the proposals can handle all the same cases with nearly identical syntax. They just pay a very small syntax tax for some cases vs the other proposals.

Specifically, F#-style is optimized solely for calling unary functions. Anything else pays a syntax tax of three characters over what Hack-style would do, with an x=> prefix to introduce the arrow-function wrapper. (This assumes the parsing issues are resolved to allow arrow-funcs as RHS without needing parens; this isn't certain to happen, as it involves some significant tradeoffs. If it doesn't happen, the tax is actually five characters, placed both before and after your code.)

Hack-style is optimized for arbitrary expressions, but if you're calling a named unary function, it pays a syntax tax of three characters over what F#-style would do, with a (#) suffix to actually invoke the function.

"Smart Mix" only pays a syntax tax (a (#) suffix, just like Hack-style) if it wants to invoke an anonymous unary function produced as the result of an expression. However, it pays a mental tax of having two separate syntax models.

...and that's it. That's all the difference is between the proposals. A three-character tax on RHSes in various situations.

My Preferences

Over time, I've become strongly in favor of Hack-style pipelines. I think that the case of "unary function" will in general be less common than "everything besides unary functions", so it makes more sense to put the tax on the first category rather than the second. (In particular, method-calling and non-unary function-calling are big cases that will never not be popular; I think those two on their own will equal or exceed the usage of unary functions, let alone including all the other syntax that Hack-style can do without a tax.)

I also think that why the tax is invoked makes Hack-style more usable; the syntax tax of Hack-style (the (#) to invoke the RHS) isn't a special case, it's just writing ordinary code in the way you normally would without a pipeline. On the other hand, F#-style requires you to distinguish between "code that resolves to an unary function" and "anything else", and remember to add the arrow-function wrapper around the latter case. val |> foo + 1 is still a syntactically valid RHS, it'll just fail at runtime because the RHS isn't callable. You can avoid having to make this recognition by always wrapping it in an arrow func, but then you're paying the tax 100% of the time and effectively just writing a slightly more verbose Hack-style.

This is also why I'm against "Smart Mix" - if you use the bare-ident F#-style case, and you realize you actually need to do anything more complicated, you need to remember to do a slight code rewrite beyond the actual change you want to make. (Tho it's a syntax error, not a runtime error, when you forget, so it's easier to catch.) You can avoid it by always writing Hack-style, but then there's no point to the mixture. (Tho at least if you do so, you're not paying the tax that F#-style does in all the other cases.)

All that said, the benefits of pipeline are so nice (again, part of the reason for jQuery's success was the syntax niceness of method chaining!) and the costs of each syntax's non-optimal case so low, that I'd happily take F#-style over nothing. So my preferences are Hack-style > Smart Mix > F#-style >>>> not doing pipeline at all.

@euan-smith
Copy link

euan-smith commented Sep 7, 2021

@benlesh absolutely good points. How would you fix async in this case? The proposal of having |> await |> doesn't seem to me to fit into what you describe either.

@theScottyJam
Copy link

theScottyJam commented Sep 7, 2021

@benlish

Some of your points you brought up against hack-style pipes are exactly the same arguments I have against F#-style pipes.

Honestly, the one of the biggest problems with the Hack-style pipeline that I see is you can't possibly create a suite of "pipeable" functions now that will work well with Hack pipeline later

One of the biggest advantages of the hack-style pipeline operator is that there is no difference between a "pipeable" function and a "non-pipeable" function. There's no such thing as making a suite of pipeable functions. The hack-style pipe works with how Javascript is used today - you can take any existing function and throw it into the pipeline operator, and it works in a very natural way.

Basically, Hack pipeline is DOA and any library that was trying to leverage functional piping will need to rewrite themselves in order to accommodate it.

And if we go with F#-style pipelines, does everyone else, standard library, third-party libraries, etc, need to rewrite themselves so that they provide curried versions of their different functions, that would fit well in the pipeline operator?

Any library out there that was helping push functional programming in JavaScript will be left swinging in the wind by JavaScript's first functional operator.

Not really. Using curried functions in a pipeline operator really isn't that bad.

const result = data
  |> transform(^)
  |> anotherTransform(^)

Is that extra (^) at the end of each step really enough to kill the usability of the pipeline operator? Perhaps it's not quite point-free programming, but it's really not that far off - you don't lose any of the advantages to point-free programming doing this. Sure, it's different and a little more verbose from other languages, but I really don't think that's killing the readability of the operator.

For example, RxJS will need to refactor all of its operators to accommodate the Hack pipeline, then everyone else will have to rewrite their RxJS code to adjust to this new paradigm.

This sounds extreme. Javascript is not a curried language, never was, and never well be. If people want to use it as one, that's fine. But, these curried libraries already know from the start that they're going against the grain. People who are using them also understand that they're going against the grain. There's always going to be a cost for doing that - e.g. whenever you want to use a non-curried function (which Javascript is full of), you'll need to transform it to a curried one first. The (^) is also just part of the cost of using Javascript in a less-standard fashion.

I don't want that last paragraph to sound too harsh - of course, it's important for the committee to be mindful of those who like to use Javascript in a curried fashion - they're part of the userbase too. But this operator is for everyone, not just the extreme functional users, and it's also important for the comittee to be mindful of everyone else who wants to use the pipeline operator without currying everything. In other words, there's many advantages that the F#-style pipe operator has over hack-style, but the argument "This is curry-friendly, at the cost of being difficult to use with non-curried functions" just does not hold a lot of water in the Javascript ecosystem.

@tabatkins
Copy link
Author

tabatkins commented Sep 7, 2021

you can't possibly create a suite of "pipeable" functions now that will work well with Hack pipeline later.

You definitely can.

If you really want to focus on maximal ease-of-use in both pipe() and |>, most libraries focused on HOFP that I've seen have an "auto-currying" function that transforms a traditionally-defined function into one that can either take all its argument at once or in multiple invocations; running this over a library will do just fine. It's not hard to write yourself, depending on how fancy you want to get; if you're just targetting "auto-curry the final argument only", it's quite straightforward. (Doing it by hand for minimal extra runtime cost is also quite easy, tho it is a little bit of boilerplate.)

But you can also just privilege one or the other, if desired, with minimal weight added on the other side:

  • you can take functions already written for pipe() and call them as pipe(foo(1,2)) or val |> foo(1,2)(^)
  • you can take functions already written for "normal" calling (such as any web platform API) and call them like pipe(x=>foo(1,2,x)) or val |> foo(1,2,^)

JavaScript's first functional operator.

Pipe isn't particular intended to be a higher-order functional operator. Some languages with it use it as such; these tend to be languages that already have heavy support for HOFP down to the syntax level (such as having functions be curried by default, as in F#), where it's a natural and obvious fit. Other langs, such as Elixir, do not - their pipe operators are still function-focused, but rely on "normal" functions rather than higher-order ones, because they just insert the piped value as an additional argument. Some languages do all of the above, like Closure, which has macros to handle piping in multiple different ways, including both HOFs and placeholders, according to the author's preference.

I pursued the pipe operator in its current form precisely because I felt that it deserved to serve more use-cases than just HOFs, because JS is not particularly oriented toward HOFP. (It allows it without too much difficulty, but lacks a lot of useful features that would make it more natural/convenient to use.) Libraries oriented strongly toward HOFP should still be able to benefit from it, as it's a generic linearization syntax that aids with lots of different code patterns, but they won't be getting maximal benefit, it's true. That's fine; the pipe() function will still exist and continue to be useful, just as much as all the other functional operators such a library is probably packing.

With the F# pipeline, any function that is currently pipeable with a standard functional pipe will "just work". Meaning zero migration for existing libraries to leverage F# pipeline OOTB.

Correct. But conversely, any function that's not currently pipeable (like every web platform API) will continue to be slightly inconvenient, meaning interacting with the rest of the web platform will continue to be slightly inconvenient. This is a "six of one, half dozen of the other" situation, where one usage pattern gets maximal convenience and other gets minor inconvenience. Figuring out the way forward is a matter of determining how to weight the two opposing classes of usage patterns so we can figure out which one can bear the burden of the minor inconvenience better. Obviously, my personal determination is that HOFs represent a lighter weight on the scale vs JS's operators, web platform APIs, and all non-HOF-based libraries. This is further influenced by the fact that HOF-based libraries already have a working and convenient pipe() function, while non-HOF-based libraries do not have a comparable linearization strategy allowed under current JS syntax without bizarre abuse of the comma operator.

(I understand that Typescript currently requires some ~shenanigans~ to make pipe() type properly, because it currently doesn't have full type unification and instead gets by with a more limited type inference strategy. It seems to be acceptable in practice, tho, and the TS team seems to be seriously looking into better inference to solve a multitude of HOFP problems similar to this.)

@tam-carre
Copy link

tam-carre commented Sep 9, 2021

To get what we want with Hack Pipeline without hamstringing the entire JavaScript functional programming community, the ideal solution (that would please everyone) is to land the F# pipeline, and then focus on the partial application proposal.

Is there a reason why this option isn't being considered?

@theScottyJam
Copy link

theScottyJam commented Sep 9, 2021

@tam-carre

It has been considered. This proposal has been around for a long time, lots of angles have been considered.

The issue with the partial application syntax is that it's very limited in its capabilities to avoid ambiguities (so no, F# + partial application will not please everyone) - you can read more about what it can and can not do in that proposal.

Here are some things that hack-style pipes can do, that partial application is simply incapable of helping out with. In other words, you would have to make a function literal in all of these scenarios if we were going with f#-style + partial application.

value |> foo(^, ^) // Using the same token twice means something different in hack-stile vs partial application
value |> f(^, g(^))
value |> ^ + 1
value |> [^, 0]
value |> {foo: ^}
value |> new f(^)
value |> `${^}`
value |> await ^
value |> (yield ^)
value |> import(^) // import() is a pseudo function, not a real one

@tam-carre
Copy link

tam-carre commented Sep 9, 2021

@theScottyJam

The usage stats for the comparison are still going to be quite significantly different if we are comparing the syntax tax for "unary functions (Hack) VS non-function-call expressions (F# with partial application)", and if we are comparing that for "unary functions (Hack) VS n+2-ary functions and non-function-call expressions (F#)", as this gist does -- which is why I'm asking why it is not currently being considered.

@sdegutis
Copy link

sdegutis commented Sep 9, 2021

value |> foo(^, ^) // Using the same token twice means something different in hack-stile vs partial application
value |> f(^, g(^))

I missed this in the proposal, and can't find it. What does each do?

@euan-smith
Copy link

euan-smith commented Sep 9, 2021

@sdegutis So the use of the partial application here is that foo(bar,?) returns a function which requires one parameter (where ? is the placeholder), so that you can do something like the hack-style with the F# proposal. In either case where one parameter of a function call is targetted val |> foo(bar,?) would do the same thing. However it does not work for a repeated parameter (or any other of the cases @theScottyJam raised above) - foo(?, bar, ?) will return a function which expects two parameters - the ? is a placeholder for any parameter, not a duplicated parameter, so it will not do the same thing as foo(^, bar, ^) in a hack-style pipe.

For the second example f(^, g(^)) would be turned into something equivalent to a=>f(a, b=>g(b) ), so the second parameter passed to the function f is a function which calls g, not the result of calling g.

@sdegutis
Copy link

sdegutis commented Sep 9, 2021

If I'm understanding you right, that's super confusing and unintuitive. I'd expect that in v |> foo(^, ^) it would assign const tmp = v and then just call foo(tmp, tmp).

@euan-smith
Copy link

euan-smith commented Sep 9, 2021

@sdegutis For the hack pipeline, you are absolutely correct. v \> foo(^, ^) would be transformed to a call to foo(v,v).

It is for the proposal to make hack-like syntax work by implementing the F# pipeline and the partial application which works differently (but looks the same).

FWIW I think the partial application proposal could cause a lot of confusion, at least initially.

@benlesh
Copy link

benlesh commented Sep 9, 2021

@tabatkins thank you for your response. If you'd like to look, I have a discussion about what RxJS will need to do if the Hack Pipeline does indeed land in JavaScript it can be found here.

If you'd please indulge me on a little bit of history around RxJS and TC39 proposals:

Since I started working on RxJS ~7 years ago in version 5 (it was going to be 3 at the time, haha), has always had to goal to move towards leveraging standards. Originally we were targeting the observable proposal and we were very hopeful about the bind :: operator. Sadly, both stalled (and are seemingly dead).

RxJS, being a once straight-port of Rx NET by Microsoft, inherited a huge assortment of methods (aka "operators" in Rx-lingo). This was, in part, because it was coming from a compiled language that is easy to optimize, and also in part because of how many operations are possible with a type like observable. The problem in JavaScript, particularly when shipping to the browser, is obviously the size that adds. So the solution was to "detach" methods and hope for that :: bind operator in the long-term, while we used prototype-patching in the short term. Prototype patching came with a litany of problems (I'm sure you can speculate here, I won't dig into that), and with :: bind basically dead, we moved to a functionally piped approach, deeply hopeful that |> would come along and give us a standards-directed way to use the library.

In short, in an effort to try to keep moving towards proposals that seem to have a solid chance we've been continuously confounded over the years. If you sense any frustration on my part, that would be the origin.

At the end of the day, my goals, and I suspect the goals of the larger RxJS team (but I won't speak for them) center around trying to deliver the best DX we can to our users.

I've been talking with @jamiebuilds, whom I honestly consider a dear friend, and whom I've been a general dick to on Twitter recently (sorry, Jamie). Where would be the appropriate place to list all of the concerns I have about the Hack Pipeline proposal concisely so they can be addressed?


Unrelated, other major issues the RxJS community cares deeply about: cancellation, any general notification APIs, and scheduling. In particular cancellation.

@sdegutis
Copy link

sdegutis commented Sep 9, 2021

These discussions and debates are helpful for clarifying to JavaScript users, to help us understand what's at stake, what the options are, and how they'll help us change our way of coding. But how can we help to push the actual process along, or cast our votes, or in any way make this become a reality? I suppose the next natural step is to use each of these options via Babel and get traction going, so that the standards committees have real world usage to go based on. But when I looked into the Babel plugin I linked to above, I can't get that plugin working in the Babel playground, which makes me think it's unsupported and that the community has moved on to another similar (or superceding) plugin. It's difficult to tell where the state of the community consensus and Babel plugins currently are.

@sdegutis
Copy link

sdegutis commented Sep 9, 2021

(@benlesh I think we asked the same basic question at roughly the same time.)

I guess a more concrete question is:

  1. Are there currently forums or mailing lists where JS end users can discuss these points with JS decision makers?
  2. If not, can we start a forum that we all agree on, and elect a moderator, or something?
  3. GitHub Issues might be a good place, especially with 👍 buttons to reduce noise. Also it's free. Also pretty sure tc39 has one.

@ljharb
Copy link

ljharb commented Sep 9, 2021

The bind operator isn’t dead; i expect a proposal to replace it to be on the next agenda. In concert with hack pipes, the bind operator and this-sensitive functions would likely work quite well together.

The locations to discuss proposals (that most people on this thread are well aware of) are a proposal’s repo, and https://es.discourse.group.

@sdegutis
Copy link

sdegutis commented Sep 9, 2021

According to a member of TC39 who I contacted, we can post these concerns directly in the TC39 repos. (I wasn't aware each proposal had their own repos.)

Which means we can post our questions and concerns in the Issues of https://github.com/tc39/proposal-pipeline-operator which is the Pipeline operator proposal, and includes all 3 options.

I'd suggest using social media to rally people to individual issues to comment or vote or try to request status updates or encourage progress or something.

@tabatkins
Copy link
Author

tabatkins commented Sep 9, 2021

Just to get ahead of any confusion, I will note that "voting" is not a signal the committee uses for anything (besides occasionally an informal temperature check on an idea). Community discussion is great for making sure we get opinions and arguments that we hadn't considered, but volume is not a factor in itself. (And, because we're humans, can be an anti-factor, if a topic induces dread due to pile-ons.)

I've also closed the majority of syntax-related issue threads in the repository now that the proposal has advanced with a chosen syntax, but I'd suggest people review some of those, particularly the longer ones, to see what has already been discussed and considered. Quite a lot of ideas have been proposed and critiqued over the three years this proposal has been developed so far.

@kiprasmel
Copy link

kiprasmel commented Sep 12, 2021

Please see tc39/proposal-pipeline-operator#91 (comment) and further discussion.

(context: the issue is for choosing which token to use for the Hack's operator, and people are struggling - I make a case that there's a reason behind it - F# + partial application > F# > no pipe operator > proposal 3 - Hack with |>> > other proposals)

Furthermore, currently the TC39 committee and their champions for the pipeline operators feel one-sided. Until we see equal representation for the F# proposal as we have for Hack from the committee (3 champions vs 0), I do not agree it was fair to reach Stage 2, let alone the possibility of moving to Stage 3.

@max-hk
Copy link

max-hk commented Sep 12, 2021

Smart Mix seems great. Was it proposed to TC39?

@lightmare
Copy link

lightmare commented Sep 12, 2021

Smart Mix seems great. Was it proposed to TC39?

Yes. The wiki points to a 2018 presentation showcasing it, and a 2021 presentation explaining that it was withdrawn in favour of Hack.

Until we see equal representation for the F# proposal as we have for Hack from the committee ...

You forgot Elixir. That one wasn't even mentioned in either of the linked presentations.

@mAAdhaTTah
Copy link

mAAdhaTTah commented Sep 12, 2021

@lightmare

You forgot Elixir. That one wasn't even mentioned in either of the linked presentations.

Elixir was considered and rejected because inserting an argument in the beginning of the argument list is very unlike JavaScript and would be confusing to developers.

@lightmare
Copy link

lightmare commented Sep 12, 2021

Elixir was considered and rejected because inserting an argument in the beginning of the argument list is very unlike JavaScript and would be confusing to developers.

That argument is about as valid as "F# calling a function without any () or `` following it is very unlike JavaScript". They're different syntaxes for a new feature, both are unlike JavaScript in a sense: one injects an argument into a function call, the other conjures up a function call where there wasn't one.

But I didn't bring up Elixir to argue its merits here. I wanted to point out that when one demands equal representation, picking a favourite proposal (which was presented twice to the committee), while completely ignoring another (which wasn't presented at all), doesn't sound much equal.

@tabatkins
Copy link
Author

tabatkins commented Sep 26, 2021

If you saw a comment from @stken2050, they're attempting to evade a ban they earned from the TC39 org for violating the CoC (being aggressive and spamming). They're now blocked from my personal account, which I believe will prevent them from commenting on my Gists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment