RubySpec is dead, long live RubySpec!

gistfile1.md

Last night, Brian Shirai unilaterally "ended" the RubySpec project, a sub-project of Rubinius (the alternative Ruby implementation which Brian was paid to work on full-time from 2007 to 2013). The blog post describing his reasons for "ending" the project led to a big discussion on Hacker News.

When a single, competing Ruby implementation tells that you its test suite is the One True Way, you should be skeptical. Charles Nutter, Ruby core committer and JRuby head honcho, spent a lot of time last night on Twitter talking to people about what this decision means. He's probably too busy and certainly too nice of a guy to write about what is a political issue in the Ruby community, so I'm going to do it on behalf of all the new or intermediate Rubyists out there that are confused by Brian's decision and what it means for us as a community.

Brian's Mission Is Good

A reference specification is a noble idea. Java has one, and it's probably a net positive for the language. If Ruby could magically have a complete implementation-agnostic language spec, today, I think everyone would be very happy with that outcome. I want to be clear that very few people in the Ruby community disagree with this idea in principle.

Practically speaking, all implementations of Ruby have absolutely benefited from RubySpec. JRuby runs RubySpec alongside the MRI suite. Many smaller implementations such as Topaz also use RubySpec. MRI uses RubySpec, most notably in the run-up to 1.9.2 (as Brian pointed out), but core members still use it today, even if it isn't officially part of the project (since all you need to do is run RubySpec against Ruby head).

Anyone that spends eight years of their life, for any reason, working in our community is someone we want to have around.

Brian's Technical Reasons Are Mostly Wrong

A lot of Brian's post is trying to spread fear, uncertainty, and doubt. Here's the statements of fact that I take issue with:

• Ruby has an ISO spec. Characterizing Ruby as a completely "spec-less" language would be incorrect. There's been an effort at an ISO spec for Ruby. Admittedly, it's not finished or perfect, and has mostly been the effort of a small group of Japanese academics. But it's there, and Matz has expressed that an ISO standard is what he sees as the way forward for a formal specification of Ruby.
• MRI's test suite has much more coverage now. It definitely used to be pretty bad, as Brian writes. The road from 1.8.7 to 1.9.3 was extremely, extremely painful for everyone, paved with bugs and minor versions. But to characterize the MRI suite as being inadequate for Ruby as a language seems inaccurate. JRuby runs almost 6000 tests and over 1 million assertions directly from the MRI suite. Does that sound like a test suite that is completely unusable to other Ruby implementations?
• MRI has used RubySpec, and some members still do. RubySpec's "minimum viable contribution" to Ruby has, and will continue to be, "does MRI pass RubySpec?". This doesn't take any official sanction or additional contribution from anyone - just run MRI against RubySpec, and open pull requests and issues when you find problems.
• A lot of MRI's process happens in Japanese. Matz, and a good chunk of the Ruby core team, is Japanese and runs on Japan time. While most of the discussion online happens in English, it can sometimes feel like there's "no process" around Ruby because a lot of discussion happens in Japanese or even in meatspace in Japan.
• Other languages use reference implementations. The absence of a formal specification, while not ideal, is not some dooming fate for a language. Python has got along just fine without one for quite a while.
• One bug doesn't change a lot. Brian pointed out that Ruby 2.2 fails a RubySpec and segfaults. This is obviously not intended - Ruby isn't supposed to segfault when running pure Ruby, ever. But one bug from a competing Ruby implementation (again, RubySpec is basically just Rubinius' test suite made re-usable) does not mean that MRI's suite is useless. All Ruby implementations have valid bug issues with MRI - and they open issues and PRs on MRI to deal with it. In fact, Brian refused to do this with this latest segfault (though he says he's opened tickets in the past). Charles Nutter had to open an bug report for him.
• Ugly tests are ugly tests, not useless ones. A lot of Brian's arguments come down to aesthetics - he thinks a lot of the MRI tests are nonsensical (some definitely are), the suite is incomplete (whose isn't?), it sets up test state inconsistently, combines too much behavior into a single test, and sometimes specify behavior that is undefined. All of these issues are valid, and they can all be fixed with a pull request. Brian's solution, instead, seems to be to throw the baby out with the bathwater and use (his) RubySpec instead.

Brian's Methods Are Controversial

I have to say that this is the tough part of this blog. Shit-slinging about personal slights and disagreements doesn't help move the issue forward, and I think the discussion about the ending of RubySpec has been very civil so far. However, there is a long history of non-technical reasons as to why Brian, Rubinius and RubySpec have been controversial in the Ruby community, and Ruby developers who haven't had time to follow the issue would be done a disservice if we all pretended that no one has personal issues with Brian Shirai.

• Brian doesn't value the contributions others have made to RubySpec. Brian doesn't think that the other committers to RubySpec have made meaningful contributions to the project. This attitude does not grow a community around an open-source project. It's actively hostile towards new contributors. It's not hard to imagine why RubySpec is struggling for contributions with a project lead that says things like:
  • "write specs in way that conforms to the standards I set"
  • "you have a few PRs, that's nice"
  • "I've had to fix a ton of crap committed by many people"
  • "ensure they abide by my quality requirements"
  • "a few PRs is not contributing"
• Brian wants more control over Ruby. For a long time, Brian has basically wanted a larger role in the decisionmaking and design process of Ruby, which, de facto, is Matz's sole privilege. This is where we get into the politics of RubySpec. Make no mistake, by making RubySpec the project (not "the test suite") "official" and requiring MRI developers to contribute to it, Brian is gaining control of the way Ruby is designed. I think it should be made clear that making RubySpec "official" in the way Brian wants is something he stands to gain from. Brian's issues with MRI's rejection of RubySpec are not technical, but political. This is OK (all large open source projects require some politicking to run), but Brian isn't making his aims and motivations clear in his blog post.
• Brian's primary responsibility is to Rubinius, not Ruby. Brian has said as much. In my opinion, he's also shown this in his actions in the past. Rubinius, his competing Ruby implementation, has shown that it's willing to break compatibility with MRI where it sees it is right to do so, or to "improve" the language in ways Brian and the Rubinius team thinks is right. This is, obviously, his perogative, but I think it's important to understand that Brian's vision of Ruby is obviously different than a lot of the core team, and Rubinius reflects this differing vision. Brian sees MRI as needing to get with his program, not the other way around.

You Can't Stop The Signal

In the end, RubySpec is open source software. It's sort of preposterous to end a project like this - Brian admits he made this decision unilaterally, and in the face of disagreement from his peers and contributors to RubySpec. Brian can't really "end" RubySpec so much as "leave the project". And it's a sad thing, I think - anyone that devotes this much of their lives to Ruby in any way is someone I wish would stay in the community. But Brian has clearly chosen to leave the mainstream of Ruby - and both MRI and Rubinius will suffer for it.

I think this whole episode carries a very poignant lesson about the way open source software is run. Brian seems to think that disagreements in OSS should be resolved with large, sweeping changes and formal governance - RubySpec and the proposed Ruby design process are concrete examples of this opinion. However, many successful projects, like Ruby and Rails, have operated with a much more piecemeal approach: incremental change, one pull request at a time.

RubySpec continues to exist, as code. Running RubySpec as a separate test suite, externally, and simply reporting the bugs back to MRI has always been and will continue to be useful. In fact, it seems like something good might come out of this episode after all - Charles Nutter (again, an MRI core committer) has discussed being open to including RubySpec in the MRI repo as a separate test suite, something which Brian has, to my knowledge, never proposed (I would guess since it would relinquish the control of the project to Matz and the Ruby core team).

Unfortunately, drive-by trolls and new Ruby developers will inevitably see this as "another sign Ruby is dying!!1". The reality is that Ruby has been doing quite well without RubySpec officially driving the design process. Ruby 2.2 is faster and more memory-efficient than ever, and a single failing RubySpec with a segfault doesn't change that.

So my advice to Ruby developers is this - let's put Ruby in Schrodinger's box. It's dead, it's alive, who cares? What I know is that Ruby is still a language that I love writing, and I still build awesome things with it. None of that has changed. And I don't think it will.

EDIT: Removed some unfair assertions about Rubinius' production-readiness, and made my point about the MRI suite's usefulness more clear.

Hi Giles!

"Brian says this, but he secretly thinks that" stuff and the "Brian is just a dick" stuff, and it's a lot easier to have a conversation.

I don't think that's fair. I think Brian has a negative attitude towards contributors, I didn't say he's a dick. We're talking about why people haven't contributed to RubySpec - Brian's attitude and demeanor towards contributors is a valid point in this discussion.

Likewise, when you've got a gist which purports to clear the air, and it states that nobody uses Rubinius in production, and the first comment is "I use Rubinius in production for serious internet business," that's not what successful attempts at clearing the air ever look like.

I'm going to remove this assertion. It's probably unfair and doesn't really have anything to do with the issue at hand.

But I also totally agree with Mr. Shirai and Mr. Peterse that the code for the MRI tests is not good. I mean they're in Mr. Shirai's rant, and Mr. Peterse linked them in his comment. It's easy to find them, but it's not easy to read them. They are certainly not perfect. And Mr. Shirai also said something about MRI being developed off "trunk," which implies either Subversion or CVS. That's not even modern technology.

I think I need to reword that bullet. What I mean is that "MRI's tests are much more complete". I agree that they're far from perfect (all of the points Brian raises about the test suites aesthetics and clarity are absolutely valid). From what I understand, Yorick and Brian think that because some (even most!) of MRI's tests are difficult to read and complex means they're useless and need to be rewritten from scrap. I disagree - coverage is coverage, IMO, even when it's not great. Charles (ha!) mentioned on Twitter that he thinks Ruby core moving to git is a possibility for 2015.

Thanks for the comment, Giles.

The elephant in the room that needs addressing is this:

https://github.com/nurse/rubyspec

rubyci runs nurse's version of rubyspec. it even accepts PRs.

So mri team were unable to work with the rubyspec team, were forced to fork rubyspec but have not been backporting specs into the branch.

One VERY constructive way of helping out here is backporting all the missing specs into this branch.

Sam, do you have any knowledge about when that was forked?

the divergence appears to date from April 23, 2013.

@nateberkopec

From what I understand, Yorick and Brian think that because some (even most!) of MRI's tests are difficult to read and complex means they're useless and need to be rewritten from scrap.

There's a risk of overgeneralization here. The word "useless" does appear in Mr. Peterse's comments, but Mr. Shirai's rant uses "inadequate." Certainly similar, but not quite the same.

Also, re "rewritten from scrap," you could pretty easily make the case that some of these tests have already been rewritten from scrap, and have been available in a rewritten-from-scrap state for eight years.

My interpretation was that the RubySpec, JRuby, and Rubinius teams have, over the years, sometimes found it very difficult to use the MRI tests. Mr. Peterse mentioned a specific situation where he and Mr. Nutter found the documentation more useful than the tests.

To me, those tests look like legacy code. I've never seen anybody who wrestled with legacy code and didn't want to just throw it all out and start again clean. This rarely goes well. I've done it successfully, once, but I've failed at it too, and I think the one success story was probably a fluke. It wasn't a gigantic project like Ruby.

Anyway, if you've got clean code and ugly code, and you've built up enough clean code to replace some of the ugly code, I think it's a good idea to do that, but I also think it's a very delicate process.

Also, I think Ruby core moving to git would be awesome.

+1 for having experienced undue difficulty getting contributions pulled into rubyspec.

OTOH - look at the bright side. currently MRI passes nearly all of the tests. Only 48 errrors or failures out of 154,917 expectations. Thanks to rubyspec, many nonworking feature are succinctly exposed. Perhaps consistent bug reports and gentle prodding might help convince the MRI core team to make future releases meet the rubyspecs.

My question is, which fork to use now?

Since I have been mentioned at least twice in regards to the latest rubyspec drama, then I will mention two things:

1. I really disliked submitting to rubyspec; so I stopped figuring working on an implementation which many people derive benefits from was enough to not be criticized.
2. If Brian wants to pull Rubyspec that is his prerogative. Someone can fork it if they want and should if they think it is important. If people look back at the last irc log where I was mentioned you will see I had this same opinion back then as well.

Notice both of these points boil down to the same thing: Don't try and force someone to do something they don't want to. My corollary to this is that people should put energies in OSS towards working things they enjoy. It ends up being a positive multiplier since others get the benefits of your work and you feel good about it.

I know this comment will be unpopular in the Ruby community, but I think it needs to be said:

As a fan of Ruby who has used it with production systems ranging from tens of millions to hundreds of millions of monthly active users, I'd like to politely suggest that the whole discussion is essentially moot until Ruby resolves perf issues. I've ported several apps from Ruby to Node with improvements ranging from 2-10x in simultaneous connection handling and 3-10x improvement in server response times. We abandoned Ruby for reasons similar to Twitter's -- incurable systemic slowness.

Obviously, some of the perf gains I have witnessed have been due to better algorithms and more mature understanding of the required system design at rewrite time, but we also ran a suite of greenfield benchmark tests with Ruby and Node configurations (optimized by a team familiar with production Ruby). Ruby got thoroughly trounced.

I know part of the Ruby philosophy is to favor developer productivity over raw system perf, but that argument flies out the window when developers have to wait frequently for specs to finish running. Ported specs ran in 30 seconds+ on Ruby and less than 3 seconds in Node. OUCH. I wish I could post the spec source. It was a horror show butchering that I'm sure better Ruby devs could have improved on.

Node is also trouncing Ruby handily on developer productivity in another way: Isomorphic code. We can run almost identical code on both the server and the client-side. AFAIK, there isn't a serious movement to use transpiled Ruby -> JS in production, so this advantage is not easily swept away (yet).

As I mentioned, I'm really a fan of Ruby. There are amazing ideas, and the syntax is beautiful, but right now, it just can't hold a candle to Node for large scale / enterprise production, and that makes me sad. I want Ruby to be a serious contender. I'm rooting for the Ruby community.

IMO, this is a much more serious battle than any argument about standard specifications. It's all moot if developers can be more productive using a directly competitive platform -- and right now, they can, and are.

There is no compelling reason for people to switch from Node to Ruby. I'd like to see the Ruby community focus on that instead of infighting over standards that won't mean a thing if Ruby continues to get left in the dust.

Programming language diversity is a great thing, and I'd really like to see Ruby have a chance in this fight. Don't waste your time getting mad at me. I stopped paying attention to Ruby after the 2nd port project because I don't have time to wait around for it. Instead, try to be open to the criticism and fix it.

I wish you all the best. I'm rooting for Ruby!

P.S. If you want to try Node but you hate JS syntax, I've used CoffeScript in large production deploys too. You'll recognize (and appreciate) some of its features.

@gilesbowkett

the divergence appears to date from April 23, 2013.

The actual day is Feb 20, 2013

Also, I think Ruby core moving to git would be awesome.

Use https://github.com/ruby/ruby

Anyway for people who want to write tests for ruby implementation, use simple test framework. Since the ruby implementation is not reliable in this case, the test framework must be simple and reliable. If you use rscec/mspec, you will see a SEGV inside of the test framework.

ericelliott:

That is just a use case for javascript. Many ruby hackers do not need javascript - that is why they can use opal.

Ruby will get faster so it won't be an issue. Guess why mruby was started in the first place? One day it will replace the old ruby, or serve as the base for which to extent into MRI.

@ericelliott Hate to say so, but your post is as off-topic as it can be for this discussion.

Hey YorickPeterse:

"If you are referring to Rubinius X, this is an idea to build what we think would be "Ruby 10", but Rubinius itself will always continue to exist."

So you are actually agreeing that you intend to do SOMETHING else other than Ruby with Rubinius, yes?

I am sorry but Rubinius changed the moment evan was gone. Today's Rubinius is different and so are its goals. The identity of a project changes the moment the key people leave - you can take _why as an example. Shoes nowadays depend on java - in the past it did not. So it is no longer the same. Nor will we see new whycats, that cartoon is gone forever.

Google used to have a "don't do evil" slogan; they abandoned it when they got bigger.

Things change.

Rubinius is already no longer having the same goals as it used to have and it will continue to diversify and change into something ... who knows. The maintainers of Rubinius will decide on that.

But it will definitely no longer be Ruby itself. You are just piggybacking on Ruby while changing Rubinius into something else. You should be more honest about the REAL goals of Rubinius too.

jdickey: "Let's have a consortium, or foundation-type org, take over maintainership of RubySpec or a fork thereof, and steer it towards something that everybody can agree is a Good Thing"

No. Why would this be needed? It seems your personal preference rather than where you actually mention why it would be required. It was not required in the past either. There is this huge flag-waving "omg omg omg we need RubySpec omg omg omg". Well we did not really, and now that it is gone we can all move on.

There is no drama - you just make it want to appear as a drama.

There is also one distinct part I disagree with nate - now I have no particular opinion pro or con with Brian; I have no problem with him at all either. For me Rubinius lost its importance when evan went away. I do know very well that several people don't get along with Brian, that is all fine. But the statement to "unilaterlly end" ... erm ... RubySpec got started by the Rubinius guys, ok? And the license allows you to take over, yes?

So what is the problem THERE please? I concur on several other parts and I don't think Rubinius will be about Ruby in the future either. Theys imply felt that their time on RubySpec was wasted, which is PERFECTLY right for them to want to do so. It is their time, too. So if you really think that RubySpec is so important or awesome, and I personally don't think it is, then you could take it over and then we will see. Otherwise it just seems unfair that you state that Brian "unilaterally closed" it ... want to take it over or what nate? ;)

Hey @shevegan,

Couple of problems with your last post.

• Rubinius will continue to support MRI Ruby semantics
• Rubinius X will be a test bed for different language features but it is a separate project
• Titanius will be the "platform" for creating languages. Rubinius and Rubinius X will use Titanius as their foundation.

This information is available in the posts that Brian put up during the month of December on the Rubinius blog. So, your statement that Rubinius will no longer be Ruby is false. It's unfortunate that these different things all have similar names, so your confusion is understandable or forgivable.

Regarding Evan, yes, the project changed when he left. Not better or worse, just different. I haven't ever seen you in the Rubinius IRC channel; did you have a lot of experience with the project at any point (either pre or post Evan)? What's your last experience with Rubinius on your system running against your code?

Regarding a consortium, it's a good idea. Right now Ruby semantics change from version to version and even from patch level to patch level. It's a moving target that no alternative implementation has been able to hit yet. Unless MRI's development process changes, it will be difficult for any team to ever have perfect compatibility with it. Even mruby (written by Matz himself) isn't compatible with MRI. If Matz can't do it, what chance do the rest of us have?

Rubyspec has been forked. We'll see if the Ruby community actually uses it and improves it. Further, it will be interesting to see if the specs remain of high quality or if it devolves into some of the junk tests that we see in MRI. I have high hopes.

Lastly, the shoes project is alive and well. It's better now than it ever was under _why. It's been refactored so that any GUI lib can be used on the back end. The first target was Java (Swing?) but someone could write for your favorite GUI lib. The code is all Ruby unlike _why's mishmash of Ruby and terrible C (seriously, check out the old C code... it's inscrutable). My point in correcting you about shoes is that projects under new ownership are not automatically worse; a new team can bring new ideas and new enthusiasm (or not). Hopefully the new owners of the forked Rubyspec kick some ass and make it great.

@shevegen,

Ruby the language should and must be separate from Ruby in any particular implementation. This would allow both for innovation and for the business folk (who pay most of our paycheques and bills) to have a far greater degree of confidence in stability and continuity than is presently justified. (The impetus for the current discussion is merely one example of that.) If the language is to be distinct from any individual implementation, then there needs to be some organised "someone" identifiable as responsible for that, and there needs to be an objective means of demonstrating conformity with a given version of the language. Hence, there is a critical need for a follow-on to RubySpec with less of the political wrangling and historical bad blood that surrounds the current project.

Saying "Matz is BDL and MRI is the reference implementation" merely identifies our single points of failure. Matz, as wonderful as he is, is one guy, and things happen to people all the time. MRI is a widely-used implementation, but it's not really organised as a reference implementation, a teaching tool, anymore (if it ever was). YARV supplanting Matz's (original/evolved) implementation and becoming "the new MRI" was just the last nail in that coffin.

Sure, it's open source; we can have fifty different repos with fifty ever-so-slightly-different versions of RubySpec. What does that gain the community beyond confusion and a growing certainty by those outside the inner-circle-of-the-moment that we as a community aren't interested in getting our shit together?

We need to grow up. We need a community that is more than the sum total of our individual egos. We need a language that teams can implement and maintain without worrying that the rug is going to be jerked out from under them, or worrying that they're going to have to spend half their total effort-hours fighting the political fires. I've been in this craft for 35 years and the only community I've seen that's this playground-ego-driven has been the BSD communities, and that's been a huge influence on the growing use and popularity of Linux. Just to make an obvious example.

It's not what I want; I didn't create (for example) rubydrama.com. I would never have even known it existed were it not for Mr Shirai's post; the fact that it does exist, and is as widely known and referenced as your search engine of choice would indicate, should be sobering. People have a tendency to identify with the work and tools that they love; that's part of what keeps us doing this job. Adults learn to work together and, especially, eventually figure out that the world isn't all about them. I don't want to elbow in and take over anything; I'm just trying to prod a piece of the discussion that, having seen similar discussions start and evolve in other communities, I'm convinced is existentially required in ours. You may disagree with me; fine. Make a better proposal, and sell it to the community at large. Threatening to hold your breath until your neck turns blue, or shouting down anyone who dares to voice ideas different than yours, undermines this community; it does not help it.

Projects that mean something should be about more than one person, no matter how talented; that's just the Beer Truck Rule in action. ("What happens if your top three guys get run over by a beer truck while coming back from lunch a week before you ship?") Saying "oh, (this project) doesn't mean anything now that (some early leader) left" is just announcing to the world that either the project is too immature and irrelevant to be taken seriously, or possibly that you are. Very likely, as is the way of the world, a bit of both. I doubt that was your intended effect.

Ruby does "mean something", to us and to a lot of other people. Shouldn't we, collectively and individually, act as though it does, and take our responsibilities more seriously?

Note that I am a very casual user of Ruby, and by no means a 'programmer'... to describe myself that way would be false advertising. Because my interest is of a totally different nature than that of real programmers, by definition I have less of a vested interest, and as such, my observations are just that - observations with no value judgement attached to the conflicting opinions posted here.

It seems to me that the issues raised here are based on unknown and incomplete information:

1. How does Matz feel about other competing implementations of Ruby - has anyone ever asked him? If he doesn't favor them, it's completely natural that he wouldn't want anything to do with anything that makes them easier to establish, as contributing to and using RubySpec would do, and quite aside from any benefits Ruby might derive from doing so.

2. How does Matz feel about the whole performance issue? Perhaps it barely qualifies as a priority for him. If he does care, perhaps he should make clear the boundaries of his concerns and what he proposes to do to address them so that people will have a better sense if Ruby is 'right' for them, and whether some fork or even a new, whole different language is the best choice for them.

As an aside, kudos to Mr. Bowkett for his emphasis on civility. All too often, although certainly not always, insulting behavior is used in an attempt to mask the faults in someone's arguments, or to flat out try and intimidate an opponent. Even when it isn't, kindergarten-style playroom behavior is often a net negative... As a case in point, it's been well documented that Microsoft's no-holds-barred embrace of this kind of culture had deeply corrosive side effects. Just because we have absolute free speech (aside from defamation) -- and should have! -- doesn't mean that there aren't things that shouldn't be said. Gratuitous insults fall into that category.