Contributing to Clojure

gistfile1.md

So you’d like to contribute to Clojure, great! Let’s talk about what that involves.

The first thing you’ll want to make sure is that your idea is valid, and that you won’t spend a ton of time working on something that won’t make into master. To do this, you should create a JIRA ticket. For example, let’s say we want to improve how core.async handles channel closing propagation. It’s not a super complex problem, but there are some design questions about which of the various semantics currently in place should be the default, and if some semantics should be configurable.

So start by making a JIRA ticket and stating what the problem is you’re trying to solve, what the possible options for solving the problem. Now hit save and wait for the ticket to be triaged. Alex Miller will take a look when he can, and that can take a few days to a few weeks, depending on the time of the year (he has other responsibilities). Alex may out-right reject the idea if he knows Rich would never approve the ticket, but otherwise he’ll probably reply fairly quickly with something to the extent of asking Rich what he thinks of the idea.

From there you’ll need to wait a bit longer until Rich can weigh-in. Rich is a mortal like all of us and so his time is limited. This means the time you’ll have to wait is unbounded. Rich works on designs for Clojure, Datomic, spec, tools.deps, core.async, and various other projects, so he’s got a lot to go over. Probably within a few weeks you’ll get an answer (via Alex most likely).

Now you have to feedback to work with: this feedback will probably include a bunch of other options or factors to consider. Take that into account in your ticket, and then fire it back of to Alex, who will talk to Rich, who will talk to Alex, who will provide more information on the ticket. Do this until everyone is happy with the plan.

Now you can write code. Write up a patch and attach it to the JIRA. The code will be looked at by Alex, probably Stu, and eventually Rich. Each of those people may provide feedback that you will need to incorporate. Eventually Rich will mark the patch as “good” and Stu will commit it to master.

So what’s the problem with this process? What’s a bit glossed over in this write-up and in the contribution process docs is the amount of time commitment involved. As I mentioned, Rich is a busy person, but he’s also what the speed of this whole process hinges on. Sure Alex optimizes this, but that’s mostly as a “no” filter, he saves Rich having to review patches that would never make it into Clojure.

Rich is a busy person, so like every human he must prioritize tickets. And as Rich mentions in his talk on “Hammock Time”, humans are limited to juggling about 5-7 “things” at a time. So if your patch is on some far-flung part of Clojure that isn’t a major priority at the moment, it will be deprioritized and will take longer for Rich to take a look at the issue. Also, at different points of time people are pulled into other work. Alex manages at least two conferences, and there are times of the year that are busy for the maintainers of Datomic. So depending on everything, it may take a few weeks to a month to get feedback on a patch you submitted.

So when I say in passing “if it doesn’t matter to Rich, it won’t get in”, it’s not a slight, it’s a statement of fact. People are limited, and must prioritize, your ticket will most likely be deprioritized unless it’s directly related to whatever Rich is currently working on.

What’s also under-mentioned is the context-switching the patch creator (you) will have to accept. During all this back and forth between Rich, Alex, and you, weeks may pass. During that time another patch may be approved that breaks your code, and you’ll have to rebase and update your patch. The time delay between your updates and getting more feedback may be weeks and months, so you’ll have to remember what you were working on, etc.

In addition make sure every assumption, every detail you’ve thought of, every possible side-effect of your code, is mentioned in the ticket. Because if you forget to mention something, Rich will most likely catch it, and hand the ticket back to you with a comment of “did you think about X”, that will add another few weeks into your dev time.

Now begins the “personal opinion” section, what I’ve stated here are the facts as I’ve worked on Clojure and the core projects. I got tired of the constant back-and-forth. Never being able to talk to the decision maker directly aside through a 3rd party. Problems that could be solved via a 10 minute meeting blow up into months of back and forth discussions, and if any party gets busy and forgets to get back to the other about the ticket, that process just takes longer. With each new project in Clojure this process just takes longer, as the less important tickets just get moved further and further into the backlog. It’s not that no-one cares about X, it’s that there’s 20 more important things and so your patch will have to wait. This results in frustration, rage quitting, and twitter wars, between people who (on all sides) have almost become too frustrated and tired to care.

So let me make this very clear...I don’t dislike Rich, Stu, or Alex, they all do great work, and do so when the rest of us are relaxing. So can we all find a better way to collaborate? With each contributor that leaves, the work on the core team just gets larger. With each new library (like spec) that requires more thought and design, the older projects must be deprioritized. And my concern is that, like a person trying to run on on the top of a lake, at some point the drag will pull us all down to a stand-still.

Isn't the overall result, that is Clojure core language ecosystem, a proof of the fact that this process.. well.. works very well.
Good noise filters are the main tools of success in modernity. And time, the process you mostly criticize here, is the ultimate filter.

Looking to other languages, the most problems are arguably created by overzealousness (yes <*>.js I am looking on you)

Timothy, thanks for taking the time to step back and write this without rancor. I know you are frustrated, and I appreciate the effort it takes to craft respectful and considered feedback.

Would you consider changing “if it doesn’t matter to Rich, it won’t get in” to "it won't get in until Rich looks at it, and you may wait a long time". The current wording implies that Rich has a casually dismissive attitude, which is unnecessarily provocative and not necessary to the point you are trying to make.

This is a pretty accurate description of the process. I believe that it works extremely well, if and once you accept that the priority is the quality of Clojure, and not the comfort of contributors. As Timothy points out, contributing to core is not fun at all:

• You have to learn a process that is probably unfamiliar.
• Most ideas will be rejected.
• Rejections may not include good explanations.
• You will have to wait, sometimes for years.
• Once an idea is on deck, the back-and-forth can still take a long time, and is optimized for Rich's time, not yours.
• Even after all this and working code, your work might still be rejected (this is very rare but has happened).

I have experienced my own share of "contributor discomfort", and I put up with it because I value the outcome. The proof is in the pudding. But if that all sounds like a punch in the face, then you should not work on core. There are many other ways to participate in the Clojure community:

• Build cool applications.
• Make libraries (Clojure is a Lisp, most of the work is in libraries anyway).
• Write documentation and tutorials.
• Teach.
• Go to conferences and user group meetings.

I know that there are people who will never agree with me. I will show those people the following respect, which I also expect from them:

• Please do not propose that I change how I work unless you honestly think you have a novel argument to make.
• Please do not accuse me of not listening to feedback. I do listen, but I may disagree and choose not to obey.
• Please do not claim to speak for the Clojure community. That is an appeal to authority, not an argument.

I would prefer discussing where work really is needed and change is cheap, with the core team's insights.

Clojure's contribution process isn't optimized for Rich's time because Rich's time is really important. It's optimized for one person so that things don't fall through the cracks. Rich happens to be a person who's good at not letting things fall through the cracks. Clojure has a clean, consistent design that is a tremendously rare sight because this has been the attitude from the beginning. Not just adding something because someone wanted it, but really considering the future effects. Common Lisp is an example of a "distributed design responsibility". I prefer Clojure.

That being said, Timothy's perspective isn't invalid. It's important. And I'd refer to his recent appearance on the defn podcast for more details.

As I see it, we should separate between future-sensitive design and plain work that could be done. I believe in keeping a core part of Clojure that is hard to move, but identify parts that would be cheap to change in the future. One example is documentation. Which I believe has been commented on by core contributers before, requesting people to write guides such as Getting Started. These changes aren't able to destroy the future, because we can change them as we go along.

I do think Cognitect could communicate this distinction clearer. I think there are tasks where the value of more work is higher than the potential cost of a bad decision. I currently spend a lot of time learning the Clojure ecosystem. It would be a valuable effort for me to produce some explanation, some documentation that is needed regardless, with some support along the way. Learning while explaining often results in a simple point of view. Stu points this out in the comment above.

That being said, I think Clojure is great, and I trust Cognitect and the community.

Teodor

FWIW, I penned a quick terrible twitter response to Tim and Stu's posts here: https://twitter.com/cemerick/status/1067111260611850240

I can empathize with Timothy and Chas here, although given I have yet to use Clojure as my primary work tool, I have never needed changes to Clojure for things related to my job to work. I have been a pretty darned active hobbyist.

Hopefully this is additional light (and not heat) for this conversation, with something Rich Hickey wrote in late 2017. Erik Assum beat me to creating an issue with the suggestion of adding this to somewhere on the Clojure site for new potential contributors to read. Quoting Rich via copy-and-paste from part of this post: https://www.reddit.com/r/Clojure/comments/73yznc/on_whose_authority/

"Clojure was not originally primarily a community effort, and it isn't primarily one now. That has to be ok. The presumption that everything is or ought to be a community endeavor is severely broken. A true community respects the autonomy of its participants, else it degenerates into a cult of need/want.

Cognitect is a terrific supporter of Clojure, and happens to be where I work, but the ultimate authority and stewardship of Clojure remains with me and I am responsible for the decisions governing it, good or bad."

The perfect Lisp doesn’t exist, and although we all have ideas what it would look like, we don't all agree. Clojure is a good-enough Lisp, which already has reasonable tooling and an excellent selection of libraries. Given that Clojure's core semantics are fairly clean and fairly well specified (as evidenced by the fact that a very great many libraries run equally on Clojure, Clojure.net and ClojureScript), if folk get really fed up with development practice within "official" Clojure, forking the project would really be no big deal.

Meanwhile, like I say, it's good enough.

See also:
https://gist.github.com/richhickey/1563cddea1002958f96e7ba9519972d9#gistcomment-2770124

Rejections may not include good explanations.

@stuarthalloway That remark puzzled me a bit. Shouldn't rejections always be explained (subjectively) clearly, so there's no bitterness left in the people who tried to contribute? I think when there's good communication and extremely clear expectations set every process works out reasonably well at the end of the day.

@bbatsov As an example of another person's take on it, the first 14 minutes or so of the talk "The Hard Parts of Open Source" by Evan Czaplicki ( https://www.youtube.com/watch?v=o_4EX4dPppA&t=485s ) were pretty enlightening to me. In particular, the part about how time consuming it can be for the project owner to give a sufficiently detailed explanation that attempts to satisfy the person making the request. Also that delegation is pretty tough to do effectively.

The same kind of conversation in golang community golang/go#21956

This is a pretty accurate description of the process. I believe that it works extremely well, if and once you accept that the priority is the quality of Clojure, and not the comfort of contributors. As Timothy points out, contributing to core is not fun at all:

• You have to learn a process that is probably unfamiliar.
• Most ideas will be rejected.
• Rejections may not include good explanations.
• You will have to wait, sometimes for years.
• Once an idea is on deck, the back-and-forth can still take a long time, and is optimized for Rich's time, not yours.
• Even after all this and working code, your work might still be rejected (this is very rare but has happened).

I have experienced my own share of "contributor discomfort", and I put up with it because I value the outcome. The proof is in the pudding. But if that all sounds like a punch in the face, then you should not work on core. There are many other ways to participate in the Clojure community:

• Build cool applications.
• Make libraries (Clojure is a Lisp, most of the work is in libraries anyway).
• Write documentation and tutorials.
• Teach.
• Go to conferences and user group meetings.

I know that there are people who will never agree with me. I will show those people the following respect, which I also expect from them:

• Please do not propose that I change how I work unless you honestly think you have a novel argument to make.
• Please do not accuse me of not listening to feedback. I do listen, but I may disagree and choose not to obey.
• Please do not claim to speak for the Clojure community. That is an appeal to authority, not an argument.