Skip to content

Instantly share code, notes, and snippets.

@tabatkins
Last active October 17, 2022 22:40
Show Gist options
  • Save tabatkins/1261b108b9e6cdab5ad5df4b8021bcb5 to your computer and use it in GitHub Desktop.
Save tabatkins/1261b108b9e6cdab5ad5df4b8021bcb5 to your computer and use it in GitHub Desktop.
Comparing the three pipeline proposals

We've been deadlocked for a while on the pipeline operator proposal, for a few reasons. Partially it's just low implementor interest, as this is fundamentally just syntactic sugar, but also because there are three competing proposals and the proponents of each haven't been convinced by the others yet.

In this essay I hope to briefly outline the problem space, summarize the three proposals, and talk about what's gained/lost by each of them. (Spoiler: they're all nearly identical; we're arguing over very small potatoes.)

What is the Pipeline Operator?

When you call a JS function on a value, there are currently two fundamental ways to do so: passing the value as an argument (nesting the functions if there are multiple calls), or calling the function as a method on the value (chaining more method calls if there are multiple).

That is, three(two(one(val))) vs val.one().two().three().

The first style, nesting, is generally applicable - it works for any function and any value. However, it's difficult to read as the nesting increases: the flow of execution moves right-to-left, rather than the left-to-right reading of normal code execution; if there are multiple arguments at some levels it even bounces back and forth, as your eyes jump right to find a function name and left to find the additional arguments; and editting the code afterwards can be fraught as you have to find the correct place to insert new arguments among many difficult-to-distinguish parens.

The second style, chaining, is only usable if the value has the functions designated as methods for its class. This limits its applicability, but when it applies, it's generally more usable and easier to read and write: execution flows left to right; all the arguments for a given function are grouped with the function name; and editting the code later to insert or delete more function calls is trivial, since you just have to put your cursor in one spot and start typing or delete one contiguous run of characters with a clear separator.

The benefits of method chaining are so attractive that some very popular libraries contort their code structure specifically to allow more method chaining. (jQuery being the elephant in the room, as it's still the most popular JS library in the world, and its core design is a single über-object with a jillion methods on it, all of which return the same object type so you can continue chaining.)

The Pipeline operator attempts to marry the convenience and ease of method chaining with the wide applicability of function nesting. The general structure of all the pipeline operators is val |> one() |> two() |> three(), where one, two, and three are all ordinary functions that take the value as an argument; the |> glyph then does some degree of magic to "pipe" the value from the LHS into the function.

The three pipeline proposals just differ slightly on what the "magic" is, and thus on precisely how you spell your code when using |>.

Proposal One: F#-style

In this proposal, matching the F# language's pipeline syntax, the RHS of the pipeline is an expression that must resolve to a function, which is then called with the LHS as its sole argument. That is, you write val |> one |> two |> three to pipe val thru the three functions; the general syntax transform is left |> right => right(left).

Pro: The restriction that the RHS must resolve to a function lets you write very terse pipelines when the operation you want to perform is already a named function.

Pro: Only one new bit of syntax needs to be minted, the |> itself. (The others require a placeholder syntax as well.)

Con: The restriction means that any operations that are performed by other syntax must be done by wrapping the operation in an arrow func: val |> x=>x[0], val |> x=>x.foo(), val |> x=>x+1, val |> x=>new Foo(x) etc. Even calling a named function requires wrapping, if you need to pass more than one argument: val |> x=>one(1, x).

Con: The yield and await keywords are scoped to their containing function, and thus can't be handled by the arrow-func workaround from the previous paragraph. If you want to integrate them into pipeline at all (rather than requiring the pipeline to be paren-wrapped and prefixed with await), they need to be handled as a special syntax case: val |> await |> one to simulate one(await val), etc.

Proposal Two: Hack-style

In this proposal, matching the Hack language's pipeline syntax, the RHS of the pipeline is an expression containing a special placeholder variable, which is evaluated with the placeholder bound to the LHS value. That is, you write val |> one(#) |> two(#) |> three(#) to pipe val thru the three functions; the general syntax transform is left |> right(#) => right(left).

Pro: The RHS can be any expression, and the placeholder can go anywhere any normal variable identifier could go, so you can pipe to any code you want without any special rules at all: val |> one(#) for functions, val |> one(1, #) for multi-arg functions, val |> #.foo() for method calls (or val |> obj.foo(#), for the other side), val |> # + 1 for math, val |> new Foo(#) for constructing, val |> await # for awaiting promises, etc.

Con: If all you're doing is piping thru already-defined unary functions, it's slightly more verbose than F#-style since you need to actually write the function call syntax, adding a (#) to it.

Proposal 3: "Smart Mix"

In this proposal, mixing the previous two proposals, the RHS of the pipeline is either an expression containing a special placeholder variable, which is evaluated with the placeholder bound to the LHS value, or a bare (dotted-)ident which must resolve to a function, which is called with the LHS as its sole argument. That is, you write val |> one |> # + 2 |> three.foo to pipe val thru the expressions; the general syntax transform is either left |> right(#) => right(left) or left |> right => right(left).

Pros: It can handle all Hack-style cases with identical syntax, and additionally can handle most F#-style cases with identical syntax.

Con: If you're doing any higher-level programming to produce a function which'll eventually get called, you do need to use the placeholder; you can't rely on F#-style evaluation. That is, if you have a logger(fn) function that modifies a function to log its args and return value, you must pipe it like val |> logger(one)(#); writing val |> logger(one) is a syntax error. (This avoids garden-path issues, where it's not clear which syntax you're dealing with until you either spot the # or establish that there isn't one. Instead, the F#-style cases are trivial to spot because they're so restricted in syntax, and all the rest are Hack-style.)

Con: As a result of trying to split the difference between the two syntaxes, it ends up with a more complex mental model. If a pipeline RHS is currently written in F#-style, and you realize you want to do something else to it, you have to recognize that it must be shifted to Hack-style syntax. (Luckily it's a syntax error if you forget, rather than just evaluating to the wrong thing, so at least it's easy to recognize after-the-fact that this is needed.)

What's The Difference?

Ultimately... very little.

As long as F#-style has the special syntax forms for await and yield, then all the proposals can handle all the same cases with nearly identical syntax. They just pay a very small syntax tax for some cases vs the other proposals.

Specifically, F#-style is optimized solely for calling unary functions. Anything else pays a syntax tax of three characters over what Hack-style would do, with an x=> prefix to introduce the arrow-function wrapper. (This assumes the parsing issues are resolved to allow arrow-funcs as RHS without needing parens; this isn't certain to happen, as it involves some significant tradeoffs. If it doesn't happen, the tax is actually five characters, placed both before and after your code.)

Hack-style is optimized for arbitrary expressions, but if you're calling a named unary function, it pays a syntax tax of three characters over what F#-style would do, with a (#) suffix to actually invoke the function.

"Smart Mix" only pays a syntax tax (a (#) suffix, just like Hack-style) if it wants to invoke an anonymous unary function produced as the result of an expression. However, it pays a mental tax of having two separate syntax models.

...and that's it. That's all the difference is between the proposals. A three-character tax on RHSes in various situations.

My Preferences

Over time, I've become strongly in favor of Hack-style pipelines. I think that the case of "unary function" will in general be less common than "everything besides unary functions", so it makes more sense to put the tax on the first category rather than the second. (In particular, method-calling and non-unary function-calling are big cases that will never not be popular; I think those two on their own will equal or exceed the usage of unary functions, let alone including all the other syntax that Hack-style can do without a tax.)

I also think that why the tax is invoked makes Hack-style more usable; the syntax tax of Hack-style (the (#) to invoke the RHS) isn't a special case, it's just writing ordinary code in the way you normally would without a pipeline. On the other hand, F#-style requires you to distinguish between "code that resolves to an unary function" and "anything else", and remember to add the arrow-function wrapper around the latter case. val |> foo + 1 is still a syntactically valid RHS, it'll just fail at runtime because the RHS isn't callable. You can avoid having to make this recognition by always wrapping it in an arrow func, but then you're paying the tax 100% of the time and effectively just writing a slightly more verbose Hack-style.

This is also why I'm against "Smart Mix" - if you use the bare-ident F#-style case, and you realize you actually need to do anything more complicated, you need to remember to do a slight code rewrite beyond the actual change you want to make. (Tho it's a syntax error, not a runtime error, when you forget, so it's easier to catch.) You can avoid it by always writing Hack-style, but then there's no point to the mixture. (Tho at least if you do so, you're not paying the tax that F#-style does in all the other cases.)

All that said, the benefits of pipeline are so nice (again, part of the reason for jQuery's success was the syntax niceness of method chaining!) and the costs of each syntax's non-optimal case so low, that I'd happily take F#-style over nothing. So my preferences are Hack-style > Smart Mix > F#-style >>>> not doing pipeline at all.

@thesoftwarephilosopher
Copy link

If I'm understanding you right, that's super confusing and unintuitive. I'd expect that in v |> foo(^, ^) it would assign const tmp = v and then just call foo(tmp, tmp).

@euan-smith
Copy link

euan-smith commented Sep 9, 2021

@sdegutis For the hack pipeline, you are absolutely correct. v \> foo(^, ^) would be transformed to a call to foo(v,v).

It is for the proposal to make hack-like syntax work by implementing the F# pipeline and the partial application which works differently (but looks the same).

FWIW I think the partial application proposal could cause a lot of confusion, at least initially.

@benlesh
Copy link

benlesh commented Sep 9, 2021

@tabatkins thank you for your response. If you'd like to look, I have a discussion about what RxJS will need to do if the Hack Pipeline does indeed land in JavaScript it can be found here.

If you'd please indulge me on a little bit of history around RxJS and TC39 proposals:

Since I started working on RxJS ~7 years ago in version 5 (it was going to be 3 at the time, haha), has always had to goal to move towards leveraging standards. Originally we were targeting the observable proposal and we were very hopeful about the bind :: operator. Sadly, both stalled (and are seemingly dead).

RxJS, being a once straight-port of Rx NET by Microsoft, inherited a huge assortment of methods (aka "operators" in Rx-lingo). This was, in part, because it was coming from a compiled language that is easy to optimize, and also in part because of how many operations are possible with a type like observable. The problem in JavaScript, particularly when shipping to the browser, is obviously the size that adds. So the solution was to "detach" methods and hope for that :: bind operator in the long-term, while we used prototype-patching in the short term. Prototype patching came with a litany of problems (I'm sure you can speculate here, I won't dig into that), and with :: bind basically dead, we moved to a functionally piped approach, deeply hopeful that |> would come along and give us a standards-directed way to use the library.

In short, in an effort to try to keep moving towards proposals that seem to have a solid chance we've been continuously confounded over the years. If you sense any frustration on my part, that would be the origin.

At the end of the day, my goals, and I suspect the goals of the larger RxJS team (but I won't speak for them) center around trying to deliver the best DX we can to our users.

I've been talking with @jamiebuilds, whom I honestly consider a dear friend, and whom I've been a general dick to on Twitter recently (sorry, Jamie). Where would be the appropriate place to list all of the concerns I have about the Hack Pipeline proposal concisely so they can be addressed?


Unrelated, other major issues the RxJS community cares deeply about: cancellation, any general notification APIs, and scheduling. In particular cancellation.

@thesoftwarephilosopher
Copy link

These discussions and debates are helpful for clarifying to JavaScript users, to help us understand what's at stake, what the options are, and how they'll help us change our way of coding. But how can we help to push the actual process along, or cast our votes, or in any way make this become a reality? I suppose the next natural step is to use each of these options via Babel and get traction going, so that the standards committees have real world usage to go based on. But when I looked into the Babel plugin I linked to above, I can't get that plugin working in the Babel playground, which makes me think it's unsupported and that the community has moved on to another similar (or superceding) plugin. It's difficult to tell where the state of the community consensus and Babel plugins currently are.

@thesoftwarephilosopher
Copy link

thesoftwarephilosopher commented Sep 9, 2021

(@benlesh I think we asked the same basic question at roughly the same time.)

I guess a more concrete question is:

  1. Are there currently forums or mailing lists where JS end users can discuss these points with JS decision makers?
  2. If not, can we start a forum that we all agree on, and elect a moderator, or something?
  3. GitHub Issues might be a good place, especially with 👍 buttons to reduce noise. Also it's free. Also pretty sure tc39 has one.

@ljharb
Copy link

ljharb commented Sep 9, 2021

The bind operator isn’t dead; i expect a proposal to replace it to be on the next agenda. In concert with hack pipes, the bind operator and this-sensitive functions would likely work quite well together.

The locations to discuss proposals (that most people on this thread are well aware of) are a proposal’s repo, and https://es.discourse.group.

@thesoftwarephilosopher
Copy link

According to a member of TC39 who I contacted, we can post these concerns directly in the TC39 repos. (I wasn't aware each proposal had their own repos.)

Which means we can post our questions and concerns in the Issues of https://github.com/tc39/proposal-pipeline-operator which is the Pipeline operator proposal, and includes all 3 options.

I'd suggest using social media to rally people to individual issues to comment or vote or try to request status updates or encourage progress or something.

@tabatkins
Copy link
Author

Just to get ahead of any confusion, I will note that "voting" is not a signal the committee uses for anything (besides occasionally an informal temperature check on an idea). Community discussion is great for making sure we get opinions and arguments that we hadn't considered, but volume is not a factor in itself. (And, because we're humans, can be an anti-factor, if a topic induces dread due to pile-ons.)

I've also closed the majority of syntax-related issue threads in the repository now that the proposal has advanced with a chosen syntax, but I'd suggest people review some of those, particularly the longer ones, to see what has already been discussed and considered. Quite a lot of ideas have been proposed and critiqued over the three years this proposal has been developed so far.

@kiprasmel
Copy link

kiprasmel commented Sep 12, 2021

Please see tc39/proposal-pipeline-operator#91 (comment) and further discussion.

(context: the issue is for choosing which token to use for the Hack's operator, and people are struggling - I make a case that there's a reason behind it - F# + partial application > F# > no pipe operator > proposal 3 - Hack with |>> > other proposals)

Furthermore, currently the TC39 committee and their champions for the pipeline operators feel one-sided. Until we see equal representation for the F# proposal as we have for Hack from the committee (3 champions vs 0), I do not agree it was fair to reach Stage 2, let alone the possibility of moving to Stage 3.

@max-hk
Copy link

max-hk commented Sep 12, 2021

Smart Mix seems great. Was it proposed to TC39?

@lightmare
Copy link

Smart Mix seems great. Was it proposed to TC39?

Yes. The wiki points to a 2018 presentation showcasing it, and a 2021 presentation explaining that it was withdrawn in favour of Hack.

Until we see equal representation for the F# proposal as we have for Hack from the committee ...

You forgot Elixir. That one wasn't even mentioned in either of the linked presentations.

@mAAdhaTTah
Copy link

@lightmare

You forgot Elixir. That one wasn't even mentioned in either of the linked presentations.

Elixir was considered and rejected because inserting an argument in the beginning of the argument list is very unlike JavaScript and would be confusing to developers.

@lightmare
Copy link

Elixir was considered and rejected because inserting an argument in the beginning of the argument list is very unlike JavaScript and would be confusing to developers.

That argument is about as valid as "F# calling a function without any () or `` following it is very unlike JavaScript". They're different syntaxes for a new feature, both are unlike JavaScript in a sense: one injects an argument into a function call, the other conjures up a function call where there wasn't one.

But I didn't bring up Elixir to argue its merits here. I wanted to point out that when one demands equal representation, picking a favourite proposal (which was presented twice to the committee), while completely ignoring another (which wasn't presented at all), doesn't sound much equal.

@tabatkins
Copy link
Author

If you saw a comment from @stken2050, they're attempting to evade a ban they earned from the TC39 org for violating the CoC (being aggressive and spamming). They're now blocked from my personal account, which I believe will prevent them from commenting on my Gists.

@Zachiah
Copy link

Zachiah commented Oct 15, 2022

What worries me about the hack-style proposal is nesting pipelines, would this just be strictly disallowed?

@ljharb
Copy link

ljharb commented Oct 15, 2022

No, as it should be unambiguous which pipeline a placeholder belongs to, and every pipeline step must use a placeholder

@Zachiah
Copy link

Zachiah commented Oct 15, 2022

@ljharb Are you allowed to use the placeholder multiple times?

@ljharb
Copy link

ljharb commented Oct 15, 2022

@Zachiah i don't believe so, no. if there's something you think won't work, please file an issue on the pipeline repo - it'll get more attention than on this gist.

@Zachiah
Copy link

Zachiah commented Oct 15, 2022

@ljharb Ok I didn't want to do something like that until I made sure I understood it fully. So to be clear, you are saying that in order to use the pipeline value multiple times the only way would be:

value |> ((v => {
    // use v multiple times here
})(%))()

This kind of defeats the point...

In this example I think the F# proposal would be a clear winner

value |> v => {
  // use v multiple times here
}

Is this not mentioned just because using the value multiple times is a rare use case or is there something I'm missing?

EDIT:

I was thinking, I guess you probably wouldn't need to use the value multiple times very often because instead, you could just add more steps in the pipeline. I guess it's hard for me to envision what a world with pipelines would be like so I'm thinking of it in terms of a world without pipelines, which is not really accurate.

@mAAdhaTTah
Copy link

I think you can use it multiple times in a single step, e.g. 2 |> % + % would be valid. There was some debate around these sorts of corner cases with multiple placeholders, esp w/ % + the modulo operator.

@tabatkins
Copy link
Author

tabatkins commented Oct 17, 2022

Yes, you can use the topic variable as many times as you want (as long as you use it at least once). It's just a variable with a special name.

@tabatkins
Copy link
Author

Nesting pipelines is also allowed, tho generally a bad idea - you should split up your code into separate statements at that point. The outer topic gets shadowed by the inner topic, so if you're directly nesting, it's equivalent to just lift the inner steps into the outer pipeline; if you're doing something more complicated that involves an inner pipeline at some point, that's fine.

That is, x |> (^^ |> ^^ + 1) |> ^^ * 2 is perfectly valid, but equivalent to just x |> ^^ + 1 |> ^^ * 2. But x |> ()=>{...something that uses a pipe...} |> ^^ + 1 is perfectly valid, but would probably be better written by splitting the statement up. (Pipeline is, after all, a tool for improving code readability; doing overly complex stuff within it defeats the point.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment