The origins of the river of CPAN (QAH 2015)

2015-01-30.md

Subject: What I want for Christm^W the QA hackathon
Date: Fri, 30 Jan 2015 12:20:46 +0100
From: Peter Rabbitson <rabbit@rabbit.us>
To: { about 35 people, various stakeholders around Perl and CPAN }

(This mail is addressed to a lot of people, mainly to the list of current participants, and then to some extra folks who I think ought to be in Berlin anyway, and even some more people on BCC. Sorry for taking your collective time)

Greetings!

I've been dithering over this email for the past... many many days. As much as I dislike myself for it, I can't find a way to not-write it. Almost all of you know me personally, thus I have great hope that the borderline stuff below will be taken at face value, instead of taking the alternative route of labeling it all "flinging poo"(sic).

On the surface we are doing great. There are amazing things like the PRC, there are various CPAN upload leaderboards and challenges. We engage with newcomers more. We have "legalized" adoption mechanisms within the PAUSE permission system. We are not dead (cue a sleepy yay).

Stepping outside of our beloved echochamber a bit however... we suck. Tangibly, measurably, throw-mugs-at-walls suck. Compile a fresh perl (or run a "cleaner" on an existing one... if you dare), and try to install something used heavily in production, like say Catalyst. Do not go for coffee in the meantime but sit through that. Consider that every single newbie ends up sitting through that, likely several times before they get the hang of it. More importantly - consider how much of the stuff scrolling by is there to actually do something for the end user. Then consider how much of the remaining stuff has unstable, unqualified, or openly hostile authorship. Then ruffle through bugreports (some of the addressed people do that anyway) - consider how many of the transient issues are caused by poisonous I know better than you developer hubris. And then how much of it is caused by cheerful wide eyed developers earning points at yet another challenge.

I find todays answers to the above "hypotheticals" appalling (and getting worse year after year). Moreover I repeatedly see a social problem underpinning a vast majority of the issues: we optimize for mutual ego-stroking, instead of code. The result is a direction of the cpan cabal that only roughly coincides with what users in soulcrushcorp cubicals and university dorms actually need. This "semi-peaceful coexistence by accident" is... it has to change.

I didn't yet put my name on the attendee list. Not because I have a scheduling conflict, but because I have a conflict of conscience. At a quality assurance hackathon I want to deal with... things that diminish the (gasp) quality of what we put out collectively as a product (volunteers or not - we are still putting out a real product, and this brings in more responsibilities than many of you want to admit). I can go into great details about what I want to tackle, but the core overriding problem is that almost none of it will require code.

It will require hours if not days of painful meticulous discussions (likely shouting matches). Discussions that will have to set aside egos, eagerness, perceived (or worse self-appointed) "rockstar problem solver" status, and similar traits. A discussion from which the only way out can be a cohesive cpan-chain body which has the mandate to do just about anything as long as they are in a near-unanimous state.

A body which can take any cpan developer (maintainer or not) and tell them "you need to cool down, let the reviewers recharge, please go do something else (not a suggestion)". A body which can take any cpan developer (maintainer or not) and tell them "your last upload - not cool, either fix every point on this list without argument, or hand the namespace over". A body that can do these things, and that will do these things when other options are exhausted. A body which is willing to do this while putting the health of the already extremely fragile ecosystem ahead of individual feelings and convictions. And a body whose members are comfortable with doing this in the open. And as I said - none of this can be solved with yet more code.

Which brings me back to why I hesitate going: "I don't want someone coming to a hackathon if it's just going to make the whole thing miserable for everyone"(sic). I can not possibly impose this discussion on people who simply do not care about it, yet we need at least 2/3rds of the attendees to participate in this, for any hope of long term success.

So... yeah, that's about all I have to say. I added an entry on the wiki (ordered by how I perceive its importance). If what I said above resonates with you, and you think such an effort has a chance - put your name on the "by" list.

And if not, well it is what it is. There will always be a long list of things to work on besides politics.

Cheers

2015-01-31.md

Subject: Re: What I want for Christm^W the QA hackathon
Date: Sat, 31 Jan 2015 06:09:38 +0100
From: Peter Rabbitson <rabbit@rabbit.us>
To: { about 35 people, various stakeholders around Perl and CPAN }

Alright, as I expected a clarification follow up is necessary. I will Reply: back to xdg as he is the only one who spoke up publicly, though I received 3 extra off-the-record replies (this is one of the issues I am trying to address btw).

All 4 replies touched on the same theme:

This is not going to happen. Perl is open. CPAN is open. I don't see any path towards the kind of "command and control" governance you describe.

Except I never described command and control, nor a body that rules and imposes stuff. What I am trying to do is establish a foundation for a set of tools that can help with solutions for the following types of (non-hypothetical) problems:

• Drastic changes of module development course by new-adopters: an established module is being given to the first-who-asks-for-it, with no recourse for what happens from this point on.

• Drastic changes of functionality for established baseline dependencies (what we loosely call toolchain) because of "X has decided to clean house"

• Single-point-of-failure authors going missing for months on end without a structure in place to do a "non-maintainer-upload".

• Frivolous deprecation of module A in lieu of module B, because of some weird back-stage (non-public) vendetta against a particular character (without a solid technical reason to do so)

• Frivolous deprecations of feature-complete modules, because a new version exists where an author wanted to experiment with reimplementing the exact same functionality in a "modern" way

• Continuing reliance on modules by first-come's who are shown again and again that they are not interested in "the common good"

To put this in even more specific perspective: In just the last year we have had issues (some luckily resolved, some not) with:

• The Alt:: debacle (n.b. essentially a documented attack on the PAUSE index)
• Carp (n.b. resolved after 6 months(!) of broken dist)
• ExtUtils::MakeMaker (n.b. too many problems to link, culminating post-writeup with emergency revert)
• List::MoreUtils (n.b. half of the issues were resolved after a 10 months long battle)
• Module::Runtime
• Moo/strictures (n.b. resolved after a 15 months long battle)
• Test::More (n.b. what was included with blead at the time was even worse than the current state)
• version.pm (n.b. as of Dec 2015 no progress has been made on the underlying cause)

I highly appreciate xdg's enthusiasm for the multi-index approach, but it has absolutely nothing to do with fixing problems on such a low level in the "dependency inverted pyramid". Because (surprise) perl doesn't have runtime namespace hierarchies. You get only one Test::More, and only one Module::Runtime, etc, etc, etc. If anything, the multi-index is a distraction from the actual problem.

Later on it was said:

Instead of a negative vision of "CPAN police" taking away namespaces

You are misrepresenting my words, I never proposed that specifically. The point I am (still) trying to get across is that all we currently have is carrots. There are no "legally defined" sticks. We need to define some sticks, even if we quietly agree to never use them. But the "writing on the wall" should be there, so that we can reduce at least by an order of magnitude the need to run around and put out localized fires.

I want us to collectively get our heads out of our asses and look at the big picture. Nobody should care if sharayanto is uploading his 1456th module, or that someone is experimenting with something on CPAN without affecting anything else. But there has to be a framework for quick and concerted pushback when the main pyramid we are referring as "the perl libraries" starts to shake. This pushback may be a joint plea to an author, a joint blogpost, a joint statement of the CPAN curators are strongly recommending avoiding X, and using Y instead because of A B and C. And yes, in the case of new adoptions resulting in abuse - a joint action to invoke a mechanism for nullification of the adoption (call it "taking away namespaces" if you will).

The key in all of the above is "joint action". You likely do not realize it, but almost every 3 weeks someone from the recipients of this mail is popping up on private chat with Did you see what X just uploaded? GROAN!. As if I as a person had some magical power of "wag of the finger". Even answers to this very email happened in private. This is ridiculous. And the fix is not more code, but more communication, and willingness to publicly stand behind the results of such communication.

This is also why I stressed that the mandate is limitless as long as it remains near unanimous. Nobody is proposing a CPAN BDFL (that would be ridiculous). The minimum amount of people involved in this is likely 10-ish. This is not a simple feel-good undertaking, it does require a lot of buy-in, and the buy-in has to come from within, from your own agreement that the current status quo is unworkable going forward.

I wasn't being overly dramatic when I said that I am considering not attending. Because if I do I will relentlessly dog each and every one of you until I get a direct answer to the above points, and get an explicit commitment or non-commitment. I am stepping up to the (considerable) pain of going through that, as the only thing I can tangibly contribute to perl today. Because no amount of code will solve the "quiet shuffling in the hallways" brand of bullshit politics we are all engaged in without anyone admitting it anyway.

But I also realize that all of the above may be my own brand of delusion. Maybe everything is just dandy, and I am the only one who sees fires across the board. If this is the case - I think I deserve to know, at the very least to save me the wasted effort of trying to get engagement on the above points in the future (both virtually and live).

If all you want to do at the QAH 2015 in Berlin is to hack quietly away at a problem you find more pertinent than this call to arms - please speak now.

Cheers

2015-02-20.md

Subject: What I will be working on during the QA hackathon
Date: Fri, 20 Feb 2015 08:05:35 +0100
From: Peter Rabbitson <rabbit@rabbit.us>
To: { about 40 people, various stakeholders around Perl and CPAN }

(This mail is a followup to a thread from 3+ weeks ago. It is re-addressed to a lot of people, mainly to the list of current participants, and then to some extra folks who I think ought to be in Berlin anyway, and even some more people on BCC)

In short: The chilling effect due to almost all luminaries who spoke up proclaiming that "everything is just dandy, and some things are never going to happen" is extraordinarily powerful. Nevertheless I have concluded that I more or less have an obligation to attend this hackathon. I feel I need to do so even in the light of very low hopes for success working against any of the issues I highlight below.

Two months from now I want to get together with other interested parties and come up with a set of procedural (in a human management sense) solutions to a number of issues. The decisive word here is "management", that is: by definition NOTHING this group comes up with can be primarily software based. Additionally whatever this group comes up with must also be "ratified" by at least several PAUSE admins and the acting Pumpking, and they need to stand behind this for years to come. Ideally I am aiming for near-unanimous admission that our current standards and practices are beyond terrible, and for a plan of action with wide buy-in that "came from within". If the discussion does not take off (or worse - is forcefully shut down), I will use my sponsored time to work on DBIx::Class instead.

See you all in Berlin

Follows a non-exhaustive list of issues I want to address in no particular order:

• Lack of direction and structure within the group, which leads to considerable concentration of control without any accountability. I will simply quote an eerily relevant essay written 45(!) years ago, and leave it at that:

  If the movement continues deliberately to not select who shall exercise power, it does not thereby abolish power. All it does is abdicate the right to demand that those who do exercise power and influence be responsible for it. If the movement continues to keep power as diffuse as possible because it knows it cannot demand responsibility from those who have it, it does prevent any group or person from totally dominating. But it simultaneously insures that the movement is as ineffective as possible. Some middle ground between domination and ineffectiveness can and must be found.

• Lost (not simply missing) control over the direction of critically-dependent-upon modules. Consider: I (ribasushi) am FIRSTCOME on the DBIx::Class namespace, for better or worse one of the "crown jewels" of CPAN. I find it exceedingly problematic (ironically I am not or at least wasn't alone), that there is nothing that prevents me from doing the following:

  • I can abruptly change the direction of the project within the same namespace, with no regard to downstream deployments
  • I can abruptly (and irrevocably) declare the project "deprecated" and refuse to upload anything under the namespace
  • I can ship a new (regardless of numbering) version with frivolous incompatible changes (the devil is in the details: "frivolous" is of course open to interpretation, the point is that this interpretation can not be at my sole discretion)
  • I can arbitrarily declare some previously working code as "too hard to support", without a clear upgrade path and/or compatibility checks (again - some code is too hard to support, but again the decision can not leave my downstream users voiceless)
  • I can raise the minimum perl version, and add heavy dependencies, and can reject patches restoring the status quo for interested parties
  • I can use my wide installbase as "alpha testers" for any sort of tooling experiments, and can declare anything that does not match my personal reading of the corresponding specifications as broken and unsupported.
  • I can ragequit and handover the FIRSTCOME to anyone of my choosing
  • I can (this one is doubly controversial) take up employment somewhere DBIC dependent and can essentially hand over my FIRSTCOME to a non-benign business interest (in some jurisdictions I would be doing so implicitly from a legal standpoint)

  The reason I said "lost control" above, is because 8 years ago we (the perl cabal at the time, not this particular group) had the vision to recognize things of importance. Here is an excerpt from http://lists.scsys.co.uk/pipermail/catalyst/2006-May/007250.html (this is also a reminder to anyone laboring under the illusion that "Perl is open, CPAN is open"):

  The core development team has recognized that as an Open Source project Catalyst is now increasingly important as a platform for business systems, and is being used for increasingly larger projects with their own planning and risk-management processes, and that Catalyst will need to be able to integrate with them at a project management level. As a result, they will be implementing a series of risk-management strategies to ensure greater certainty and continuity for business users (and thus of course providing the same for all users).

  I would love to take part in some sort of social contract and enter some sort of an agreement with a "CPAN governance body", that prevents me from doing any (and more) of the things I described above. I am well aware that such tools do not currently exist within perl, and I have no problem being a "trailblazer project" to hammer these out. What I fear is being the only such project, due to severe lack of buy-in (at which point why bother?)

• Lack of distinction between a true FIRSTCOME and an adoption. While the issues outlined above can be only opt-in for primary authors, it is absurd to have them remain optional for authors adopting already established modules. Stepping further back I can not follow the logic of the adoption process being one way - i.e. the protections of a FIRSTCOME (i.e. one can throw out other comaints, the namespace is granted essentially for life, etc) somehow stretch to something that is essentially given to "the first warm body that asked for it". I do not (yet) have a good solution for this, but something is long overdue given the semi-aggressive list and the current view on adoptions within p5p. Note that my DBIC FIRSTCOME is also an adoption (handed over by mst in 2011), hence my own work falls under this category.

• Worrying trend of equating "working together" with "being nice to each other", with a wide range of implications. From the (earlier described) phenomenon of people widely agreeing on a problematic course but being afraid to speak up, all the way to active pushback against any criticism regardless of whether it has technical merit or not. Just several months ago I was "shamed" on a private thread for "abusing my community standing to discredit a module" (In my eye I was (and still am) performing my civic duty as an engineer, recommending users stay away from a piece of software with questionable install- and/or run-time practices). I feel that this feedback loop needs to be addressed in less uncertain terms.

• Tangible radicalization of authors within various parts of the dependency chain. This isn't even a "problem" as such, but is a worrying observation, especially combined with the rest of the issues I brought up. Here is a short list of some anonymous quotes (written, typos included, in public channels within the last 12 months), to illustrate my point (Yes, I know all about taking things out of context. My point is that the trend is there, and to the best of my knowledge none of the quotes were within a joke-setting):

  • does the toolchain have to support windows? *sigh*
  • when you want to use CPAN, use a recent (5.14+) - otherwise please do "perl Makefile.PL; make"
  • I am tired fo tracking down odd behavioral things on 5.8
  • I'm sick and tired of all of the bitching about version objects. I was going to ignore this thread because I'm completely burned out with p5p in general
  • we all know the world needs to be dragged kicking and screaming
  • I have just always assumed there was no reasont o not want the latest and greatest of anything, so I have just always let cpan give me the latest and greatest, and I never questioned it or looked back at what older ones had to offer. I jsut always assume the newer is better/faster/harder/stronger
  • the way i see this toolchain thing is, play the big game, take the big risks

  Contrast the above with the (a bit vague but still) perlpolicy.

2015-02-21.md

Subject: Re: What I will be working on during the QA hackathon
Date: Sat, 21 Feb 2015 13:14:24 +0100
From: Peter Rabbitson <rabbit@rabbit.us>
To: { about 40 people, various stakeholders around Perl and CPAN }

On 02/20/2015 01:07 PM, withheld wrote:

On Fri, Feb 20, 2015 at 08:05:35AM +0100, Peter Rabbitson wrote:

I would love to take part in some sort of social contract and enter some sort of an agreement with a "CPAN governance body", that prevents me from doing any (and more) of the things I described above.

Can your argument be shortened to "successful distributions" do not belong to their author(s) any more, and must be overseen by the "CPAN governance body"?

In the real practical world - the answer is sadly but firmly no [1]. This, however, is not the crux of my argument.

I think people keep misunderstanding the fine line I am trying to draw here for the two types of FIRSTCOME. Think of CPAN permission on a specific module as US citizenship**[2]**.

1. Once you are "natural born" you just are, and there is not much (if anything) that can be done to take this away from you**[3]**.

This is how I see "real first come". An extension of this is the Catalyst situation (see further comment below) - I consider the version of events known to me a black mark in PAUSE history, and will go above and beyond to add procedures preventing this from ever happening again.

For the cases of "real first come" a hypothetical body can do nothing beyond "we recommend X be avoided in lieu of the alternative Y". In case this sounds preposterous, just consider JSON::XS vs JSON::MaybeXS, or Path::Class vs Path::Tiny, and how nobody (apart from Marc and Ken themselves [4]) has a problem with this.

The contention point is that even this ability (in my eye - duty) is currently contested. To reiterate the tired example: I recommended in a public channel that people avoid a distribution and (still) haven't heard the end of it. This is fucked up.

Moreover there is no mechanism of becoming a "model citizen", which I outlined earlier as "social contract". My understanding is that there is extremely low interest in something like this, while at the same time it would require a certain level of buy-in to be effective in any way. I would love to be proven wrong.

2. The other side of the coin is the "naturalized citizen" or an "adoptive first come". You still have the same rights (and responsibilities), but your citizenship can be taken away from you at any point in your life. The most notable cases of US denaturalization involve persons concealing their participation in war crimes, but in practical terms any naturalized citizen turned federal felon is at risk of waking up without that citizenship.

The contention point here is that we do not make a distinction on PAUSE between the two, and the problem will only become more pronounced the more we push the ADOPTME and HANDOVER angles. Note: I am not advocating**[5]** that we retroactively take away Test::More or List::MoreUtils or something like that. These must be grandfathered**[6]** in any future plan, and the last resort for any intractable problems can be no more than "Avoid X".

What I am trying to get to is: what we have currently wrt adoptions sucks bordering on negligence, and there needs to be something that prevents such situations from ever happening again in the future. Both in terms of "critical infrastructure may not be adopted by fresh blood" and in terms of "critical adopted infrastructure may not significantly change course down the road".

Here is an excerpt from http://lists.scsys.co.uk/pipermail/catalyst/2006-May/007250.html (this is also a reminder to anyone laboring under the illusion that "Perl is open, CPAN is open"):

The core development team has recognized that as an Open Source project Catalyst is now increasingly important as a platform for business systems, and is being used for increasingly larger projects with their own planning and risk-management processes, and that Catalyst will need to be able to integrate with them at a project management level. As a result, they will be implementing a series of risk-management strategies to ensure greater certainty and continuity for business users (and thus of course providing the same for all users).

The difference with the Catalyst project is that the decision to add more rules/governance came from within. Except with people who already agree with the concept, I don't see how some external group imposing its authority can ever be welcome.

That's an interesting interpretation of what happened back then, which does not agree with anything I know about it (i.e. in my version SRI was given the Yaser Hamdi treatment**[7]**). Given the multiple interpretation exist (which is a surprise to me), I think it will not be constructive to speculate further on that precedent.

** Worrying trend of equating "working together" with "being nice to each other", with a wide range of implications. From the (earlier described) phenomenon of people widely agreeing on a problematic course but being afraid to speak up, all the way to active pushback against any criticism regardless of whether it has technical merit or not. Just several months ago I was "shamed" on a private thread for "abusing my community standing to discredit a module"[9]. I feel that this feedback loop needs to be addressed in less uncertain terms.

Isn't the issue not with "being nice" in itself, but with "be nice" being used as an agurment to silence technical criticism?

I consider this simplification misleading, hence can't answer the question as posed. The issue to me runs much deeper, and the part that you singled out is one of the least significant ones. This is a whole another topic in itself, which I think is best left until Berlin-time.

Also, if people are "afraid to speak up", isn't it because the group they're trying to address is not nice to begin with?

In my experience the #1 (by a wide margin) reason for people to not speak up is that they think they are alone. If you know you have "numbers" you are not nearly as concerned about an "angry mob of nice people". Of course the fear of the mob is the very reason people remain in isolation, reinforcing the entire loop. I do not yet know how to address this

Cheers

[1] In a perfect world - absolute, resounding, unequivocal YES: the "price" for success must be inevitably diminished control

[2] US citizenship, because that's the one I am most familiar with legally and precedent-wise.

[3] Unless you are Snowden, in which case you are cast into limbo of your citizenship still being there, but all proof of it is rendered invalid. Or unless you are Yaser Esam Hamdi in which case you are locked up with no charge under whatever current "Dictator-feel-good-act" until you give up your citizenship "voluntarily"

[4] Well, almost nobody - I for one have great reservations about both cases, but that is a discussion for another era

[5] When I am wearing my real practical world hat that is

[6] Test::More being doubly special due to dual-life status

[7] http://en.wikipedia.org/wiki/Yaser_Esam_Hamdi#Release

listening.jpg